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This  repon  discusses  processes  of  solving  an  ill-structured  problem.  We  asked  individuals 
to  design  instruction  in  a  domain  in  which  they  were  relatively  unfamiliar.  The  goal  of  this  activity 
was  relatively  open-ended,  and  a  solution  of  the  problem  required  significant  construction  of  what 
the  various  components  of  the  problem  were,  as  well  as  construction  of  solutions. 

Our  study  contributes  to  a  growing  body  of  analyses  in  cognitive  science  of  relatively  ill- 
structured  problems,  especially  in  design  tasks.  Reitman  (1965)  ^scussed  a  protocol  recorded  as  a 
musical  composer  worked  on  a  fugue.  Simon  (1973)  discussed  a  social  process  of  deciding  on 
features  of  a  sailing  ship.  Getzels  and  Csikszentmihalyi  (1976)  studied  art  students  composing 
drawings.  Several  investigators  have  studied  composition  of  written  essays  (e.g.,  Bereiter  & 
Scardamalia,  1987;  Hayes  &  Flower,  1980).  Design  of  experiments  in  microbiology  was  modeled 
by  Stefik  (1981).  Jeffries,  Turner,  Poison  and  Atwood  (1981)  studied  design  of  software.  A 
problem  of  designing  an  ad*^  nistrative-political  policy  was  studied  by  Voss  and  his  associates 
(Voss,  Greene,  &  Penner,  i  >3).  Kant  and  her  associates  (Kant,  1985;  Kant  &  Newell,  1983) 
studied  and  modeled  design  of  an  algorithm.  Design  of  a  residential  building  was  studied  by  Akin 
(1984).  Allen  (1988)  studied  graphics  designers  working  on  posters.  Ullman,  Dietterich,  and 
Stauffer  (1988)  studied  design  problem  solving  in  mechanical  engineering.  Pirolli  and  Berger 
(1991)  studied  instructional  designing  by  experienced  professionals,  and  Goel  and  Pirolli  (1991) 
presented  a  general  discussion  of  design  problem  solving  using  the  idea  of  a  design  problem  space. 

Two  general  features  characterize  problem  solving  in  these  various  design  contexts.  First, 
design  problem  spaces  are  functionally  diverse,  involving  a  variety  of  goals  and  constraints  that 
have  to  be  met  simultaneously.  Second,  design  problem  solving  activity  is  constructive,  in  that 
subgoals  and  materials  for  solving  the  problem  have  to  be  generated  by  the  problem  solver,  rather 
than  being  given  as  part  of  the  problem. 

Functional  diversity  and  constructive  activity  arc  matters  of  degree.  We  take  the  view  here 
that  design  problem  solving  can  be  analyzed  by  extending  concepts  of  information  processing  that 
have  been  used  successfully  in  analyzing  well-defined  problems  (Newell  &  Simon,  1972). 

Indeed,  solution  of  well-defined  problen  such  as  geometry  proof  exercises  can  include 
constructive  processes  and  interaction  of  multiple  problem  spaces  (Greeno,  Magone  &  Chaiklin, 
1979).  Our  discussion  here  contributes  a  discussion  of  design  problem  solving  that  is  grounded  in 
information-processing  concepts  and  thus  emphasizes  theoretical  continuity  of  design  problems 
with  other,  more  well-defined,  tasks. 


Instructional  Design 

A  task  in  instructional  design  typically  involves  specification  of  a  topic  and  an  audience  for 
whom  the  instruction  is  intended.  The  designer  then  constructs  a  plan  (at  some  level  of  detail)  that 
specifies  a  sequence  of  events,  sometimes  called  instructional  transactions,  involving  various 
subtopics  and  various  instructional  activities.  Gnstructional  activities  often  are  presentations  of 
information  by  a  teacher,  but  also  can  include  discussions  or  other  activities  by  students.) 

Activities  of  instructional  design  are  influenced  by  two  kinds  of  domain-specific 
knowledge.  One  is  detailed  knowledge  and  experience  of  teaching  in  the  subject-matter  domain  of 
the  instruction;  the  other  is  knowledge  and  experience  of  procedures  of  designing  instructional 
materials. 

Experts  in  an  instructional  domain  include  professional  workers  in  the  domain  (e.g., 
mathematicians  for  instruction  in  mathematics,  writers  for  instruction  in  literature,  or  auto 
mechanics  for  instruction  in  car  repair)  and  people  who  have  significant  experience  in  teaching  in 
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the  domain.  The  instructional  designs  constructed  by  subject-matter  experts  reflect  their  rich 
understanding  of  the  organization  of  concepts  and  principles  of  the  domain.  The  instructional 
designs  constructed  by  expert  teachers  (often  as  lesson  plans)  reflect  their  rich  experience  of 
teaching  the  material  and  fmding  ways  to  present  information  and  engage  students  that  do  or  do  not 
result  in  students'  learning. 

The  large  literature  on  instructional  design  is  mainly  prescriptive,  consisting  of  heuristic 
principles  that  can  be  used  to  organize  the  design  process  (e.g.,  Reigeluth,  1983).  These 
principles  correspond  to  a  complex  set  of  domain-specific  methods  and  strategies  in  the  practices  of 
many  experience  instructional  designers  (Pirolli  &  Berger,  1991). 

Our  research,  on  the  other  hand,  examined  design  problem-solving  activity  of  relative 
novices.  Our  designers  were  familiar  with  the  activities  of  teaching,  but  were  asked  to  design 
instruction  with  material  that  they  had  just  learned.  They  had  supervised  practice  in  constructing 
lesson  plans,  but  did  not  have  specific  courses  or  work  experience  in  instructional  design. 


Empirical  Study^ 


METHODS 

Participants.  Panicipants  were  eight  students  in  the  Stanford  Teacher  Education  Program 
(STEP).  Four  of  the  participants  had  recently  graduated  from  the  program,  and  four  were  new 
students  at  the  beginning  of  the  one-year  program.  Within  each  group  of  four  participants,  one 
male  and  one  female  student  were  planning  to  teach  high  school  science,  and  the  other  male  and 
female  students  were  planning  to  teach  either  high  school  mathematics  or  .social  studies. 

Panicipants  were  recruited  through  an  announcement  in  one  of  their  classes  and  were  paid  for  their 
participation. 

Subject-Matter.  Tne  experiments  used  a  fictional  device  as  the  topic  for  instruction.  The 
device,  a  fictional  vehicle  called  the  VST  2(X)0,  was  developed  in  previous  research  by  Greeno  and 
Berger  (1987;  1990).  The  VST  2000  has  alternative  sources  of  energy,  which  are  displayed  on  a 
computer  screen  along  with  displays  of  switches  that  can  manipulated  using  a  mouse.  By 
changing  switch  settings,  connections  between  different  components  and  states  of  the  components 
are  changed,  resulting  in  simulated  operation  of  the  vehicle  with  its  different  sources  of  energy. 

The  display  that  the  participants  interacted  with  is  shown  in  Figure  1 .  (The  components  of  the 
fictional  device  are  analogous  to  components  of  a  stereo  system:  the  solar  pack  is  like  a  radio 
receiver,  the  tablograph  is  like  a  turntable,  and  the  vegetor  is  like  a  cassette  player-recorder.  This 
analogy  was  not  mentioned  to  our  participants,  and  none  of  them  indicated  ^at  they  recognized  the 
analogy.) 

The  domain  of  this  fictitious  device  is  advantageous  for  three  reasons;  (1)  it  is  of 
manageable  size  and  complexity  so  that  a  detailed  representation  of  knowledge  about  the  device  can 
be  specified;  (2)  participants'  knowledge  about  the  device  can  be  controlled  to  a  great  extent 
because  it  is  not  a  subject  matter  that  our  participants  have  studied  previously;  and  (3)  a  computer- 
based  display  and  simulation  were  available  for  use  in  the  research. 

Procedure.  Each  participant  took  part  in  two  one-  to  two-hour  sessions  on  separate  days. 
On  the  first  day,  participants  learned  about  operating  the  VST2000  using  a  computer-based  tutorial. 
On  the  second  day,  they  were  asked  to  think  aloud  as  they  designed  two  pieces  of  instruction  about 
the  vehicle. 


^  A  more  complete  description  of  this  study  is  in  Greeno,  Korpi,  Jackson,  &  Michalchik  (1990). 
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On  the  first  day,  participants  went  through  a  computer-based  tutorial  for  the  VST2000.  In 
the  tutorial,  participants  read  text  interspersed  with  13  multiple-choice  questions  about  the 
VST2000,  its  components,  and  the  functional  relations  among  the  components.  Some  of  the 
questions  required  the  participant  to  manipulate  the  switches  on  the  VST2000  to  simulate  the 
operation  of  the  device.  The  computer-based  tutorial  program  repeated  instructional  text  when  a 
participant  gave  an  incorrect  answer.  Participants  were  instructed  to  think  aloud  as  they  answered 
the  questions,  and  their  verbalizations  were  recorded  on  audio  tape.  An  interviewer  operated  the 
tape  recorder,  reminded  participants  to  think  aloud  while  answering  the  questions,  and  conducted  a 
brief  interview  at  the  end  about  the  participant's  college  major  and  background  in  science  and 
technology. 

On  the  second  day,  the  participants  were  given  training  in  verbalizing  their  thoughts. 
Following  a  procedure  adapted  by  Korpi  (1988)  from  Ericsson  and  Simon  (1984),  participants 
were  instructed  to  think  aloud  while  solving  a  series  of  problems.  When  the  participant  said  that  he 
or  she  was  comfortable  with  the  thinking-aloud  procedure  he  or  she  was  asked  to  design 
instruction  for  one  of  the  two  goals  described  below.  After  completing  the  first  design,  the 
participant  was  asked  to  design  instruction  for  the  second  goal.  The  order  in  which  the  tasks  were 
presented  was  balanced  across  the  pamcipants. 

For  one  piece  of  instruction,  called  operations,  participants  were  asked  to  design  materials 
that  a  teacher  could  use  to  instruct  high-school-aged  students  in  the  operation  of  the  VST2()00. 

The  participants  were  told  to  assume  that  the  VST20(X)  was  a  real  machine  and  that  they  should 
design  a  general  plan  to  use  in  teaching  its  operation. 

For  the  other  piece  of  instruction,  called  principles,  participants  were  asked  to  design 
materials  that  a  teacher  could  use  with  high-school-aged  students  in  a  general  science  course.  The 
VST2()0U  was  to  be  used  to  illustrate  general  principles  about  science  and  machines.  The 
participants  were  given  a  card  that  described  some  general  principles  that  they  might  address  in 
their  design.  These  were:  storing  energy,  extracting  stored  energy,  converting  energy, 
transporting  energy,  and  purifying  energy. 

During  work  on  the  design,  the  participant  was  permitted  to  refer  to  the  VST2000  screen 
display,  to  write  notes,  and  to  ask  the  experimenter  questions.  After  finishing  each  piece  of 
instruction,  the  participant  was  asked  to  review  the  design  in  relation  to  six  aspects  of  instruction: 
main  topics,  sequencing,  methods  of  presentation,  materials,  activities,  and  questions  or  problems 
for  students  to  work  on. 

After  finishing  both  tasks,  the  participant  was  asked  about  his  or  her  past  experiences  with 
leaching  and  instructional  design  to  comment  on  the  tutorial. 


RESULTS 

Transcriptions  of  the  audio-taped  records  of  each  participant's  design  activity  were  coded 
regarding  three  aspects  of  problem  solving:  subproblems,  types  of  knowing,  and  operators. 

Subproblems.  Subproblems  are  activities  that  contribute  to  satisfying  requirements  of  a 
design.  An  instructional  design  includes  specifications  of  content  to  be  included,  types  of 
instructional  transactions  that  should  occur,  a  sequence  in  which  the  material  and  transactions  will 
occur,  and  other  properties.  Designers  also  specified  or  clarified  requirements  of  the  design  task, 
clarified  aspects  of  the  subject  matter,  and  engaged  in  planning  and  monitoring  of  their  progress. 
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Figxire  2  shows  the  distribution  of  activity  for  the  10  subproblems  that  were  coded.  The 
abbreviations  along  the  abscissa  stand  for  the  following  subproblem  labels;  (1)  determine  content, 
(2)  determine  sequence,  (3)  determine  timing  (durations),  (4)  determine  instructional  transactions, 
(5)  determine  instructional  resources,  (6)  determine  constraints  or  requirements  of  the  design  task, 
(7)  introduce  own  constraints,  (8)  monitor  progress  on  the  design  or  set  out  a  plan  for  the  design 
process,  (9)  clarify  the  subject  matter  to  be  taught,  and  (10)  non-design  activities  related  to  the 
instruction.  The  quantitative  measure  was  based  on  the  transcripts;  we  measured  the  lengths  of 
segments  that  were  coded  in  each  subproblem.  The  unit  of  len^,  1  cm,  corresponded  to  about 
six  characters,  or  slightly  more  than  one  word.  Many  segments  of  the  transcripts  were  coded  as 
contributing  to  more  than  one  of  the  subproblems. 


All  Tasks 


Note  that  by  far  the  greatest  amount  of  activity  involved  specifying  what  would  be  taught  - 
the  content  ~  and  how  it  would  be  taught  -  the  transactions.  The  sequence  of  the  instruction  was 
determined  incidentally.  The  sequence  of  events  was  considered  many  times  -  this  subproblem 
had  the  highest  frequency  of  occurrence  -  but  almost  always  in  very  brief  references. 

The  general  pattern  of  Figure  2  was  quite  consistent  across  all  the  subsets  of  our  data. 
(Graphs  of  these  comparisons  are  included  >n  Greeno  et  al.,  1990.)  Comparing  the  Principles  and 
Gyrations  goals,  there  was  some  more  work  on  determining  content  in  the  Principles  designs  than 
in  the  Operations  designs,  but  Determine  Content  had  the  greatest  amount  of  activity  in  bo^  sets  of 
transcripts.  There  was  also  about  twice  as  much  activity  of  Monitoring  in  the  Principles  designs  as 
there  was  in  the  Operations  designs.  Comparing  the  STEP  graduates  with  beginning  students,  the 
main  difference  was  that  the  graduates’  protocols  were  somewhat  longer,  but  the  profiles  of  the 
two  groups  were  very  similar.  Comparing  the  designs  that  participants  did  first  and  second  in  the 
sequence,  the  second  designs  were  somewhat  longer,  especially  in  the  activity  for  determining  the 
content  and  transactions.  Participants  were  more  active  in  introducing  constraints  and  clarifying 
subject-matter  issues  in  their  second  designs,  and  attended  less  to  considering  instructional 
resources,  than  they  were  in  their  first  designs. 
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We  constructed  graphs  of  the  subproblem  activity  longitudinally  across  the  transcripts  for 
the  eight  designers  on  each  of  their  two  designs.  (These  graphs  are  included  in  Greeno  et  al., 
1990.)  Two  patterns  occurred  quite  fr^uendy.  In  the  most  frequent  pattern  (six  of  the  16 
designs,  with  three  others  that  were  quite  similar),  the  designer  spent  one  or  two  short  episodes  at 
the  beginning  clarifying  the  task.  The  main  part  of  each  design  has  a  major  emphasis  on  Determine 
Content,  with  brief  bits  of  Determine  S^uence  within  each  episode.  The  designers  occasionally 
specified  instructional  transactions  or,  less  often,  instructional  resources,  while  proposing  the 
content  Monitoring  occurred  quite  regularly  throughout  the  design  work. 

A  second  pattern  that  occurred  frequendy  (five  of  the  16  designs)  showed  about  an  equal 
balance  of  activity  on  the  Determine  Content  and  Determine  Transactions  subproblems  throughout 
work  on  the  design.  Instructional  transactions  and  content  usually  were  specified  in  separate 
episodes,  while  in  the  first  pattern  these  were  transactions  and  content  were  usually  discussed 
together. 

Types  of  knowing.  We  coded  different  types  of  knowing  according  to  inferences  that  we 
made  about  different  kinds  of  information  that  designers  drew  on  in  their  design  activities.  These 
would  correspond  to  different  constituents  of  a  knowledge  base  for  design  problem  solving.  We 
identified  three  general  groupings:  subject-matter  content  knowing,  pedagogical  knowing,  and 
pedagogical  content  knowing. 


All  Tasks 


Figure  3  shows  the  distribution  of  activity  across  all  the  designs  and  designers  of  1 1  types 
of  knowing  that  we  coded.  The  abbreviations  across  the  abscissa  stand  for:  (1)  information  about 
the  VST2000,  (2)  general  information  about  science,  (3)  general  information  about  teaching.  (4) 
general  information  about  learning,  (5)  information  about  available  resources,  (6)  information 
about  students,  (7)  information  about  learning  the  specific  material  in  the  instruction,  (8) 
uiformation  that  is  explicitly  about  teaching  the  material  in  the  instruction,  (9)  information  that  is 
implicitly  about  teaching  the  material  in  the  instruction,  usually  in  the  form  of  a  statement  that  some 
material  should  be  included  or  that  some  kind  of  transaction  should  be  used,  (10)  information 
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about  the  design  task,  and  (1 1)  personal  information,  such  as  an  experience  that  the  designer  drew 
on  or  a  personal  preference  about  a  way  of  teaching.  The  quantitative  measure  for  types  of 
knowing  was  the  same  as  for  subproblems,  and  multiple  code-categories  were  applied  to  many 
segments  of  the  transcripts.  ^ 

The  predominance  of  information  that  involved  teaching  about  specific  materials  or  with 
specific  Ipds  of  transactions  corresponds  to  the  predominance  of  work  on  the  subproblems  of 
determining  the  content  and  transactions  of  instmetion.  In  comparisons  between  the  two 
instructional  tasks,  more  of  the  activity  in  the  Operations  designs  involved  information  about  the 
VST2(XX)  and  more  of  the  activity  in  the  Principles  designs  was  about  general  information  in 
Kience,  ^  should  have  been  the  case.  The  Operations  designs  referred  somewhat  more  to  specific 
information  about  learrung  the  instrucdorial  marerial,  perhaps  because  die  designers  had  just 
learned  this  material  themselves.  The  Principles  desi^  included  more  activity  involving  implicit 
informatioti  about  te^hing  the  materials  and  transactions  to  be  included  in  the  instruction.  There 
were  no  evident  differences  between  STEP  graduates  and  beginning  students  in  their  profiles 
across  the  types  of  knowing,  other  than  the  overall  greater  length  of  the  graduates'  transcripts. 

The  typical  longitudinal  pattern  of  inferred  types  of  knowing  showed  concentrations  of  use 
of  subject-matter  content  and  specific  information  about  materials  and  transactions  throughout  the 
process.  Some  designers  were  concerned  with  general  subject-matter  information  primarily  at  the 
beginning  of  their  work  on  the  problem,  and  some  designers  focussed  on  characteristics  of  the 
design  task  for  some  time  at  the  beginning  of  their  worL 

QwratiQPS-  We  cod^  protocols  for  five  operators:  (1)  proposing  new  material  for  the 
design,  (2)  modifying  material  that  was  in  the  design,  (3)  removing  some  materia]  from  the  design, 
(4)  including  new  information  in  the  design  space,  and  (5)  commenting  on  the  design  by 
recapping,  reflecting,  evaluating,  monitoring,  or  justifying  material  in  the  design. 


I  I  ^ - 1 - 1 - 

Propose  Modify  Remove  1.1.  RREMJ 

Operator 


Figure  4.  Lengths  of  transcriptions  for  operators. 

The  distribution  of  lengths  of  protocol  segments  is  shown  in  Figure  4.  Most  distributions 
for  subsets  of  the  data  were  similar  to  Figt^e  4.  One  interesting  variation  was  that  the  Propose 
operator  took  up  somewhat  less  of  the  design  activity  in  the  Operations  task  than  the  Principles, 
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task,  and  the  metadesign  operators  RREMJ  were  used  somewhat  more  for  the  Operations  task  than 
the  Principles  task.  In  the  longitudinal  patterns,  some  designers  used  the  Propose  operator  almost 
exclusively  in  the  initial  design  and  used  RREMJ  primarily  when  they  reviewed  the  design,  while 
other  designers  interspersed  Propose  and  RREMJ  during  their  initial  design  work. 


A  Model  of  Design  Problem  Solving 

The  rest  of  this  paper  reports  results  of  an  extended  examination  of  four  of  the  protocols 
obtained  in  the  empirical  study,  focused  on  the  organization  of  problem-solving  activity.  TTie 
model  that  we  present  is  similar  to  models  developed  previously  for  design  problem  solving, 
notably  by  Hayes-Roth  and  Hayes-Roth  (1978),  by  Simon  (1973),  and  by  Stefik  (1981).  The 
design  problem  space  is  characterized  in  terms  of  several  levels  of  generality,  and  different 
functions  are  performed  by  distinguished  hypothetical  agents.  These  features  are  consistent  with 
AI  systems  that  have  been  developed  for  composing  and  planning  instruction  ,  such  as  the 
Instructional  Design  Environment  (Russell,  Moran,  &  Jordan,  1988),  the  Self-Improving 
Instructional  Planner  (Macmillan,  Emme,  &  Berkowitz,  1988),  and  ^e  Blackboard  Instructional 
Planner  (Murray,  1988). 

The  model  describes  a  Hypothetical  design  "committee"  that  can  perform  the  range  of 
activities  in  which  a  human  designer  engages.  It  organizes  and  makes  sense  of  these  activities  in 
lernis  of  a  structured  set  of  "actors"  and  their  roles.  This  committee  framework  is  flexible  in  that  it 
allows  the  addiuon  or  deletion  of  members  as  needed  to  account  for  new  data.  This  flexibility  can 
be  used  to  account  both  for  stylistic  differences  between  designers  and  for  characteristic  differences 
between  various  types  of  design  tasks,  for  example,  it  might  apply  to  instructional,  architectural  or 
engineering  design. 

We  used  tlie  model  to  interpret  think-aloud  protocol  data  in  temis  of  a  set  of  categories  for 
reducing  the  data  to  a  more  manageable  form.  The  reduced  data  constitute  a  hypothesis  about  the 
organization  of  the  designers'  activities,  the  problem  space  in  which  they  were  working  at  any 
given  time,  the  relative  dominance  of  each  type  of  actor,  and  the  order  of  activities.  The  reduced 
data  also  can  facilitate  comparative  analy.se.s,  for  example,  to  show  how  designers  differ  from  each 
other,  or  to  identify  additional  actors  and  actions  that  are  not  included  in  the  present  model 
description,  but  which  an  individual  designer  might  employ. 

The  model  was  developed  by  analyzing  think-aloud  protocols  collected  from  four  of  the 
teacher- trainee  designers  who  participated  in  our  study  as  they  each  designed  a  piece  of  instruction. 
Two  of  the  designers  were  just  beginning  their  year-long  course  of  study  and  two  had  just 
completed  it,  two  were  male  and  two  were  female,  two  were  planning  to  be  science  teachers  and 
two  were  teachers  in  other  subjects,  and  two  designers  work^  on  the  design  tasks  in  each  of  the 
two  sequences  that  we  used  in  the  study.  The  four  protocols  reflect  a  variety  of  approaches,  and 
were  selected  to  provide  a  range  of  data  on  which  to  develop  the  model. 

OVERVIEW  OF  THE  MODEL 

As  described  previously,  we  see  the  primary  task  in  design  is  one  of  construction  rather 
than  search.  We  identify  two  aspects  of  construction:  to  build  the  design  itself,  and  to  build  the 
environment,  or  problem-space,  in  which  the  design  is  constructed.  Our  model  begins  with  two 
problem- spaces  to  accommodate  these  two  aspects  of  the  task. 

We  conceptualize  the  system  that  creates  the  design  as  a  committee  of  planners,  each 
serving  particular  functions.  Figure  5  depicts  these  planners  and  their  associated  problem  spaces. 
This  design  committee  comprises  three  subgroups:  senior  managers,  middle  managers  and 
builders.  The  senior  managers  work  on  the  task  of  problem-space  construction.  They  determine 
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how  to  go  about  creating  the  design,  provide  information  necessary  for  design  construction,  and 
monitor  the  design  process.  The  middle  managers  guide  the  builders  in  constructing  the  actual 
design  by  coordinating  the  steps,  advising  on  the  design,  and  suggesting  design  possibilities.  The 
builders  are  responsible  for  putting  the  actual  pieces  into  the  design  at  various  levels  of  abstraction. 


f - 

Problem-space 

Design 

construction 

construction 

space 

space 

Middle  Managers: 

Builders: 

Monitor 

Facilitator 

Abstract 

Architect 

Advisor 

Intermediate 

Memory 

Speculator 

Detail 

« 

Secretary 

J 

Figure  5.  Design  problem  spaces  and  their  associated  actors. 

Within  each  subgroup,  various  actors  fill  the  different  roles.  The  three  senior  managers  are 
the  Monitor,  the  Architect,  and  the  Memory.  The  three  middle  managers  are  the  Facilitator,  the 
Advisor,  and  the  Speculator.  The  builders  are  arbitrarily  divided  into  three  levels  of  abstraction: 
abstract,  intermediate,  and  detail.  In  addition,  a  Secretary'  may  record  the  decisions  that  the 
builders  make.  The  next  section  describes  the  specific  functions  that  these  actors  serves.  Note  that 
the  model  includes  the  actors  and  activities  needed  to  describe  the  work  of  all  four  designers  in  our 
subsample.  Individual  designers  did  not  necessarily  employ  every  actor,  and  different  designers 
might  emphasize  different  functions  of  a  particular  actor. 

ROLES  OF  ACTORS  ON  THE  DESIGN  COMMITTEE 

Table  1  lists  the  design  committee  members  and  the  responsibilities  of  each  one.  Several  of 
the  actors  serve  more  than  one  function.  These  functions  are  distinguished  '  :  order  to  characterize 
the  general  contributions  of  each  actor,  but  they  may  overlap.  The  important  distinction  is  between 
the  functions  of  different  actors,  not  between  the  functions  of  a  single  actor. 

The  roles  of  committee  members  are  described  in  detail,  with  specific  examples,  below.  In 
the  interests  of  conciseness,  the  examples  are  presented  with  minimal  context.  We  sought 
instances  that  can  stand  alone,  but  recognize  that  there  is  some  inherent  ambiguity  in 
decontextuaUzed  statements. 

Senior  Managers 

The  senior  managers  do  not  work  on  the  design  itself.  They  provide  information  and 
perform  tasks  that  are  prerequisite  to  the  artifact  construction  woric  of  the  middle  managers  and 
builders.  In  trying  to  determine  if  a  statement  is  made  by  a  senior  manager  or  by  some  other  actor, 
the  key  question  is:  does  the  statement  contribute  directly  to  the  artifact  or  not?  If  it  does  nor,  then 
the  statement  was  made  by  a  senior  manager. 
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TABLE  1 

Design  Problem  Spaces,  Associated  Design  Committee  Members,  and  Their  Functions 
Problem-space  Construction  Space 
Senior  Managers:  determine  how  to  go  about  creating  design 

Monitor  1)  guides  design  process;  determines  procedures;  decides  next  planning  task 

2)  keeps  track  of  (monitors)  design  process 

Memory  •  spontaneously  activates  relevant  memories 

Architect  1)  identifies  constraints  &  goals  given  by  E;  defines  task 

2)  adds  or  infers  new  constraints  &  goals 

3)  identifies  or  looks  for  missing  constraints,  goals  or  information 

Design  Construction  Space 

Middle  Managers:  assist  builders  in  creating  the  design 

Facilitator  1 )  decides  what  part  of  design  to  work  on.  and  whether  to  stop  or  continue; 

directs  attention  to  different  parts  of  design 

2)  looks  for  ideas  of  what  to  include;  seeks  next  piece  of  design 

3)  keeps  track  of  (reviews)  the  state  of  the  design 

4)  adds  to  design  agenda 

Advisor  1 )  supplies  (pedagogical)  heuristics,  beliefs,  and  suggestions 

2)  explains  rationale,  purpose,  or  effect  of  design  decisions 

3)  assesses  goodness  of  design 

4)  gives  practical  advice  or  comments  on  the  design 

Speculator  •  considers  possibilities  for  the  design  w/o  including  them 

Builders:  construct  the  artifact 

Abstract  •  determines  the  general  approach  to  be  taken  (on  large  or  small  scale) 

Intermediate  •  decides  topics;  decides  how  long  topics  should  take 

Detail  I )  includes  details  into  the  design; 

2)  describes  classroom  interactions,  scenarios,  or  specific  words  to  be  used 

Secretary  •  records  design  decisions  that  have  already  been  made 

(if  a  decision  and  recording  occur  simultaneously,  the  decision  takes 
precedence,  and  the  action  is  attributed  to  one  of  the  builders  above) 
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Monitor.  The  monitor  both  guides  and  monitors  the  design  process.  Notice  that  these  jobs 
have  to  do  with  the  process  of  design,  and  not  with  the  desi^  artifact  itself.  (Compare  the 
functions  of  the  facilitator,  described  below,  which  include  similar  activities  aimed  at  the  design 
artifact) 

Guiding  the  process  may  involve  determining  the  procedures  that  will  be  used  or  deciding 
on  the  next  planning  task.  Some  examples  of  guiding  the  design  process  are:  "I’m  going  to 
brainstorm  some  things,  then  decide  how  long  i  want  this  unit  to  be  and  how  many  days  it  would 
cover,"  "Fm  going  to  list  the  objectives;"  and  "I’ll  put  the  amount  of  time  I  would  spend  on  each 
topic  in  parentheses."  Note  that  theses  examples  plan  future  actions  that  will  contribute  to  the 
design  artifact  Those  actions  themselves  will  be  performed,  not  by  a  senior  manager,  but  by 
either  a  middle  manager  or  builder. 

Monitoring  the  design  process  involves  keeping  track  of  what  one  is  doing.  Some 
examples  of  monitoring  are:  "I’m  going  back  and  forth  in  explaining  this  right  now;’’  ’Tm  stuck 
here;"  "I’m  starting  to  think  on  too  many  different  levels  right  now;"  and  "I  have  to  remember 
some  stuff  that  I've  learned." 

Architect.  The  architect  plays  a  key  role  in  addressing  the  ill-structured  nature  of  the  design 
problem  by  adding  structure  to  the  task.  This  actor  helps  to  shape  the  design  construction  space  by- 
introducing  necessary  information,  including  the  goals  and  constraints  under  which  the  design  will 
be  built.  The  architect  identifies  goals  or  constraints  included  in  the  task  assignment  (which,  in  this 
case,  would  be  design  specifications  that  the  researcher  provided),  infers  or  assumes  new  goals  or 
constraints,  and  recognizes  and  seeks  out  information  that  is  missing  from  the  design  space. 

Identifying  goals  or  constraints  given  in  the  assignment  usually  consists  of  repeating  some 
task  specification,  such  as:  "So  I'm  designing  or  outlining  an  entire  unit?"  and  "So,  [I'm  to  design] 
a  unit  of  instruction  that  can  cover  any  number  or  days." 

The  most  interesting  of  the  architect's  jobs  is  to  add  new  goals  and  constraints  to  the 
problem  space.  This  is  the  activip'  during  which  the  architect  adds  structure  to  the  ill-structured 
task.  These  constraints  may  be  added  by  inference  or  assumption.  Some  examples  are:  "I'm 
assuming  these  will  be  50-minute  classes;"  "my  assumption  is  that  the  last  unit  was  already  on 
physics,  so  they  already  know  about  simple  circuits;"  "I’m  interested  in  their  developing  the 
thinking  processes;"  and  "I’m  sure  the  Exploratorium  has  some  [exhibit]  like  that" 

The  architect  also  recognizes  when  necessary  information  is  missing  from  the  problem 
space  and  tries  to  find  it.  Missing  information  may  be  from  a  variety  of  domains,  e.g.,  design 
goals,  content,  available  resources,  and  general  knowledge.  An  example  from  each  of  these 
domains  follows  (the  questions  may  be  addressed  to  the  self  or  to  the  researcher):  "what  is  the 
bottom  point  of  this?  Am  I  trying  to  get  them  to  think  critically  and  figure  this  out  on  their  own  . . 

.,  or  do  I  just  want  them  to  be  able  to  operate  this?"  "I  can’t  remember  what  the  other  term  was 
here.  Local  energy  and  . . . ;"  "Could  I  use  a  computer  tutorial  such  as  this?"  and  "I’m  . . .  trying 
to  think  of  forms  of  energy  used  in  the  world  and  trying  ro  see  what's  missing  from  the  machine" 

Memory.  The  memory  adds  relevant,  though  not  necessarily  essential,  information  which 
the  middle  managers  and  builders  can  utilize.  The  information  is  provided  spontaneously  from  the 
store  of  all  information  in  long  term  memory.  Some  examples  are:  "the  tutorial  took  me  close  to 
two  hours;’’  "when  I  was  first  reading  this  [tutorial],  I  was  a  little  bit  confused  because  it  looked 
like  there  were  four  items  here.  It  took  me  a  while  ...  to  see  that  there  were  three  main  branches 
of  energy  going  into  one  motor,"  and  "since  I'm  not  a  science  teacher.  I'm  not  sure  what  is 
available  in  a  typical  high  school." 
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Middle  Manasgrs 

Middle  managers  provide  information,  direction,  and  suggestions  that  directly  assist  in 
creating  the  design,  without  adding  to  the  artifact  itself.  One  might  distinguish  their  work  from  that 
of  the  builders  by  imagining  the  artifact  as  a  physical  object,  consisting  of  components  that  have 
been  added  on.  For  any  statement  that  concerns  the  artifact,  the  key  question  is:  did  the  statement 
add  a  piece  to  the  artifact?  If  it  did  not,  it  was  made  by  a  middle  manager. 

Facilitator.  The  facilitator  coordinates  the  work  of  the  other  designers.  It  has  four 
functions:  to  decide  what  part  of  the  design  the  group  will  work  on;  to  look  for  ideas  of  what  to 
include  in  the  design;  to  keep  track  of  the  state  of  the  design;  and  to  add  to  the  design  agenda  (i.e., 
to  decide  what  things  the  builders  will  need  to  include). 

In  deciding  what  part  of  the  design  to  work  on,  the  facilitator  directs  the  attention  of  the 
other  designers  and  decides  whether  to  stop  or  continue  work  on  a  particular  piece  of  the  design. 
Here  arc  some  examples:  "Okay,  next  topic;"  "I'm  going  to  leave  that  for  right  now  and  go  into 
what  I  want  to  do  with  the  latter  pan  of  the  unit;"  "Okay,  now  I’m  looking  back  at  your  principles 
card;"  and  'Tm  going  to  erase  what  I  put  for  kinetic,  because i  want  to  expand  a  little  on  potential." 

The  following  are  some  examples  that  involve  looking  for  ideas  of  what  to  include  in  the 
design  (the  questions  are  addressed  to  the  self):  "Okay,  now  where  do  1  go?"  "How  long  should  1 
make  this  unit?"  "How  would  1  break  this  down  to  teach  something  like  that?”  "I'm  looking  at  the 
diagram  now  to  see  where  I  would  go  next. . .  I'm  looking  to  see  if  there  is  any  particular  topic 
that  would  make  the  most  sense  to  go  to  next:"  and  "I'm  trying  to  think  of  how  they're  going  to 
show  how  well  they've  learned  that." 

Keeping  track  of  the  state  of  the  design  involves  reviewing  the  design  as  it  presently  exists. 
(Compare  the  Monitor's  function  of  keeping  track  of  the  design  process.)  Some  examples  are: 
"I'm  looking  at  the  days:  Wednesday.  Thursday.  Friday.  I’ve  got  three  days  here;"  "so.  we've 
defined  things;  we've  motivated  them;  we've  given  the  functions  of  each  of  the  three  systems,  and 
we're  starting  to  focus  on  the  solar  energy  source  . . . ;"  and  "we're  almost  done  with  the  second 
week." 


Adding  to  the  design  agenda  is  often  associated  in  the  data  with  reviewing  the  state  of  the 
design.  In  our  data,  there  was  little  agenda-setting,  but  occasionally  the  designer  mentioned 
something  that  should  be  included.  Some  examples  follow  (agenda-setting  statements  are  printed 
in  italics):  'I  haven't  told  them  about  those  [forms  of  energy]  ycuwhich  I  would  have  to  do; "  "I 
forgot  to  talk  about ...  the  startup  switch  . . .  I’m  going  to  have  to  explain  what  that  is;" 
and” things  like  what  is  a  converter,  what  is  a  selector  switch,  what  is  a  purifier . . .  those  kinds  of 
basic  things  are  going  to  have  to  be  explained  to  them.” 

Advisor.  The  advisor  provides  the  reasoning  behind  design  decisions.  Like  the  facilitator, 
it  has  four  functions:  to  supply  heuristics,  beliefs  and  suggestions  (which,  in  the  case  of 
instructional  design,  would  have  pedagogical  content);  to  explain  the  rationale,  purpose,  or  effect 
of  design  decisions;  to  evaluate  the  design;  and  to  provide  practical  advice  or  comments.  In 
practice,  these  functions  may  overlap,  and  an  individual  statement  may  subsume  more  than  one 
function. 

In  providing  pedagogical  heuristics,  beliefs,  or  suggestions,  the  advisor  supplies  the 
philosophical  bases  for  making  design  decisions.  TTiese  statements  reflect  the  designer's 
understanding  of  how  instruction  ought  to  be  accomplished.  Some  examples  are;  "[an  advanced 
organizer]  would  be  a  very  important  thing  for  them  to  use;"  "to  be  able  to  instruct  someone  else 
means  that  they  [the  students]  really  have  to  understand  it  themselves;"  "that  might  stick  in  their 
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minds  a  little  bit  more  if  they  can  have  a  concrete  example;"  and  "I  think  that  it's  important  for 
them  to  understand  how  it's  flowing  first  I  think  they  [would]  get  more  out  of . . .  each  one  . . . 
if  they  knew  what  it  was  doing  and  where  in  the  circuitry  it's  doing  it" 

The  advisor  also  explains  the  rationale  for  desi^  decisions  or  speculations.  (In  the 
following  examples,  explanatory  statements  are  in  italics.)  "I'm  contemplating  whether  it'd  be 
worth  it  to  demonstrate  it  to  begin  vnih,just  to  get  them  to  see  what  it's  doing  before  I  start 
describing  it;"  and  "I  would  use  that  to  motivate  them,  so  that  they  could  understand  not  only  what 
is  a  motor,  but  where  does  it  fit  into  their  lives:"  and  "111  have  them ...  do  some  kind  of  activity 
before  I  go  on  to  the  next  one,  to  make  sure  they’re  with  me  and  they  don’t  lose  interest." 

The  advisor  acts  as  a  critic  in  assessing  the  goodness  of  the  design.  It  may  comment  on 
aspects  that  are  satisfactory  or  ones  that  ought  to  be  changed:  "Now  it  looks  perfect-one  day  for 
each  of  these— because  that's  one  week;"  "that’s  going  to  make  it  unrealistic  to  split  the  groups  for 
that  long;"  and  "I'm  feeling  like  all  I'm  doing  now  is  lecturing  . . .  and  I'm  afraid  I'm  going  to  lose 
them  if  I  spend  the  whole  hour  just  lecturing." 

Finally,  the  advisor  may  provide  practical  advice  or  comments  on  the  design.  For  example: 
"[the  decision]  would  depend  on  how  many  computers  were  available;"  "the  first  illustration  will 
take  a  little  bit  longer  [to  explain]  because  we'll  be  getting  into  specifics  that  they  can  relate  to  in  the 
next  two;"  "I  think  some  of  this  is  Just  going  to  take  patience  and  understanding  because  this  is  a 
complicated  thing;"  and  "this  really  lends  itself  well  to  a  discussion  of  the  different  forms  of  energy 
as  proposed  by  the  machine." 

Speculator.  The  speculator  considers  possibilities  for  what  to  include  in  the  design.  These 
suggestions  may  be  accepted,  rejected,  or  ignored  by  the  builders,  who  make  the  final  decisions 
about  what  becomes  part  of  the  artifact  Speculator’s  statements  offer  pieces  for  the  design,  and  so 
can  look  like  builders'  statements.  In  such  cases,  the  context  provides  information  about  whether 
the  piece  was  merely  suggested  or  actually  included  in  the  artifact  Some  examples  of  speculations 
are:  "you  could  build  a  simple  ramp  and  . . .  maybe  use  matchbox  cars  . . . ;"  "I'm  thinking  now 
about  whether  I  should  tell  them  about  the  differences  between  the  wires;"  "it  would  be  possible  to 
do  some  sort  of  advanced  organizer  before  using  the  tutorial;"  and  "I  wonder  if  I  told  them  to  work 
on  the  model  on  Friday,  if  it  would  be  realistic  to  have  them  do  that  over  the  week-end  and  have  it 
ready  on  Monday." 

MLdsis 

The  main  task  of  the  builders  is  to  construct  the  design  artifact  In  the  process,  they  use 
information  provided  by  the  other  actors.  We  define  the  builders  on  three  levels  of  abstraction: 
abstract  intermediate,  and  detail.  Because  the  builders  deal  at  the  level  of  the  artifact  the  tasks  and 
examples  we  ascribe  to  the  three  types  of  builders  are  specific  to  instructional  design.  In  another 
domain  of  design  (e.g.,  architecture),  one  would  expect  that  the  distinction  between  levels  of 
abstraction  would  still  hold,  but  that  the  specific  activities  would  be  different  The  decision  to 
describe  three  (versus,  say,  two  or  four)  levels  of  abstraction  is  arbitrary.  We  chose  three  levels 
because  they  fit  the  data  reasonably  weU  and  seems  to  characterize  the  design  artifact  To 
characterize  different  data  or  another  level  of  tuning,  one  might  choose  more  or  fewer  levels  of 
abstraction. 

In  addition  to  the  builders,  there  may  be  a  secretary  who  records  the  decisions  of  the 
builders,  but  who  does  not  suggest  new  pieces  for  the  design.  In  our  physical  analogy  of  the 
artifact  as  object,  the  secret^  would  do  the  work  of  carrying  the  components  from  the  builders 
and  placing  them  on  the  artifact. 
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Abstract  builder.  The  abstract  builder  determines  the  general  approach  the  instruction  will 
take,  for  example:  "I'm  going  to  follow  the  model  of  how  I  learned  it;"  "I'd  start  by  following  the 
format  of  the  review  box;"  or  "I'm  going  to  break  the  information  down  into  some  basic  categories 
and  definitions." 

Intermediate  builder.  The  intermediate  builder  lists  such  things  as  topics,  timing,  and 
activities,  without  going  into  detail.  Most  planning  was  presented  on  this  level.  For  example,  "on 
Monday,  I'd  review  the  basic  circuits  and  circuit  diagrams;"  "The  next  logical  place  to  go  after 
sources  of  energy  is  forms  of  energy;"  and  "now  that  I’ve  gone  through  one,  I  would  elaborate  on 
each  of  the  components- what  each  is  and  what  it  does."  Some  of  these  statements  may  sound  like 
planning  statements  of  the  Facilitator.  Here,  as  in  other  cases,  the  context  provides  information  as 
to  whether  or  not  the  statement  actually  added  to  the  artifact 

Detail  builder.  The  detail  builder  elaborates  fine  points  of  the  design.  For  example:  "Ill 
talk  about . . .  this  power  source  . . .  and  [how]  when  the  sun  is  shining,  the  energy  is  then  taken 
in  through  the  photo  receptor  cells,  into  the  solar  pack,  and  then  it  goes  out  the  converter  [and]  into 
the  selector  switch;"  "I'll  use  the  example  of  a  solar  calculator,"  and  "it  might  be  nice  to  ask 
anybody  if  they  would  like  to  take  that  as  their  project-to  go  to  the  Exploratorium  and  give  an  oral 
report  on  the  different  types  of  things  they  learned  about  different  types  of  energy."  The  detail 
builder  may  get  so  specific  as  to  describe  anticipated  classroom  interactions  and  the  acaial  words 
the  instructor  or  students  might  say,  for  example:  "maybe  demonstrate  rolling  the  ball  down  and 
hitting  something  else  and  say,  'how  many  other  ways  can  you  see  potential  energy  turn  into 
kinetic  energy?'  And  they  could  come  up  with  ways  I’m  sure  that  I  would  never  think  of-for 
example,  instead  of  hitting  something,  they  might  put  some  type  of  spinner  that  sits  over  the 
track. " 


Secretary.  The  secretary  does  not  actually  build  the  design,  but  ratlier  records  decisions 
that  have  been  made  by  one  of  the  above  builders.  This  role  is  included  in  the  model  to  cover  the 
single  case  in  which  a  designer  described  writing  down  design  decisions.  Examples  from  the 
secretary  included:  "so  first  I’ll  write,  'introduce  machine,  describe  parts;"’  and  "so  I’ll  note 
'Exploratorium'  here."  Occasionally,  a  decision  was  made  and  written  simultaneoush .  in  which 
case  we  attribute  the  action  to  one  of  the  above  builders  rather  than  to  the  secretary. 


APPLICATION  OF  THE  MODEL 

In  applying  the  model  to  verbal  protocol  data,  it  is  helpful  to  follow  a  structured  sequence 
of  steps.  We  mentioned  some  of  these  in  the  previous  section.  This  section  summarizes  an 
organized  set  of  guidelines  for  interpreting  data  in  terms  of  the  model. 

The  first  step  in  applying  the  model  is  to  break  the  protocol  into  units  that  represent  single 
actions.  We  refer  to  these  units  as  statements.  The  initial  segmentation  requires  judgment  on  the 
part  of  the  analyst  as  to  what  constitutes  an  action;  these  judgments  can  be  modified  later  as 
application  of  the  model  proceeds. 

The  next  step  is  to  assign  an  actor  to  each  statement  This  assignment  is  facilitated  by 
asking  a  series  of  general  questions  that  narrow  the  options  first  to  a  particular  category  of  actor, 
then  to  a  specific  actor.  These  questions  arc  listed  in  Table  2. 
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TABLE 2 

Sequence  of  Questions  for  Assigning  Verbal  Protocol  Statements  to  Actors 


1)  Does  the  statement  serve  to  define  the  problem-space  or  to  construct  the 
design? 

2a)  If  to  define  the  problem-space,  which  of  the  three  senior  managers  is  acting? 
2b)  If  to  construct  the  design,  does  the  statement  actually  contribute  to  the  design 
artifact? 

3a)  If  yes  to  2b,  on  what  level  of  abstraction  is  the  builder  contributing? 

3b)  If  no  to  2b,  which  Middle  Manager  is  acting? 


The  first  question  aims  to  determine  which  problem  space  each  statement  comes  from- 
problem-space  construction  or  design  construction.  According  to  the  model,  any  statement  that  is 
relevant  to  the  task  belongs  to  one  or  the  other  of  these  spaces.  If  the  statement  serves  to  defme  the 
problem  space,  then  the  analyst  can  identify  which  senior  manager  is  acting  by  referring  to  the 
roles  listed  in  Table  1.  If  the  statement  serves  to  construct  the  design,  the  analyst  should  ask  if  it 
actually  contributes  to  the  design  artifact  itself,  or  if  it  is  peripheral.  If  the  statement  is  pan  of  the 
artifact,  then  its  level  of  abstraction  indicates  which  builder  is  involved.  If  the  statement  is 
peripheral,  then  the  analyst  can  identify  which  middle  manager  is  acting,  again,  by  referring  to 
Table  1.  To  help  confirm  the  assignment  of  an  actor,  if  the  actor  has  more  than  one  action  listed  in 
Table  1 ,  the  analyst  should  determine  which  specific  action  is  being  performed. 

AN  ILLUSTRATION  OF  THE  MODEL  USING  A  VERBAL  PROTOCOL 

What  does  the  model  look  like  in  practice?  Below,  we  apply  the  model  to  an  excerpt  from 
the  protocol  of  a  novice  designer  we  will  call  Sally.  Sally  had  been  in  the  teacher  training  program 
for  about  one  month.  She  was  creating  a  design  to  teach  high  school  students  to  operate  the  VST. 
Table  3  shows  the  actors  who  participated  in  Ae  first  part  of  Sally's  design.  Senior  managers  are 
indicated  in  bold-face,  middle  managers  in  plain  typeface,  and  builders  in  italics.  The  actions  each 
participant  performed  can  be  determined  by  referring  to  Table  1.  In  the  case  of  actors  who  can 
perform  several  functions,  the  action  number  in  the  third  column  of  Table  3  refers  to  a  numbered 
procedure  from  Table  1. 

According  to  this  interpretation,  Sally  began  die  task  by  attempting  to  construct  the  artifacL, 
and  immediately  realized  that  she  needed  more  information  about  the  constraints  on  the  task.  Once 
she  determined  these  constraints,  she  set  off  constructing  the  design  again,  with  some  suggestions 
firom  the  middle  managers. 

In  applying  the  model  to  Sally's  entire  design,  we  found  that  the  model  provides 
interpretations  for  the  variety  of  actions  in  which  she  engaged.  To  illustrate  this  fact.  Appendix  I 
presents  the  first  half  of  Sally's  design,  analyzed  according  to  the  actors  who  participate  and  the 
specific  actions  they  perform.  We  will  refer  to  this  section  of  Sally's  protocol  again  later  in  the 
paper. 
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TABLES 


Actors  Participating  in  First  Part  of  Sally's  Design 


# 

Statement 

Actor  (Action  #) 

1 

How  would  I  teeak  it  down  to  teach  something  like  that? 

Facilitator  (2) 

2 

...  are  they  going  to  use  the  computer  at  all,  or ...  are  they 
going  to  be  learning  to  operate  this?  The  same  thing  I  did.  It's 
not  the  actual- 

Architect  (3) 

4 

Then  I  just  orally  tell  you  how  I'd  set  this 

Architect  (1) 

5 

I  guess  it  doesn't  matter  what  I  tell  you,  but  it's  good  to— 

Architect  (1) 

7 

I'm  used  to  doing  a  presentation  after  I've  thought  about  it 

Memory 

9 

I  think  what  I  would  do  first  is  give  them  a  general  overview  of 
what  it  is. 

Intennediate  Builder 

10 

Basically  sort  of  follow  how  I  learned  to  do  it.  I  guess. 

Abstract  Builder 

1! 

Tell  them  . . .  what  it  does,  the  general  purpo.se  of  it.  and  why 
it's  unique. 

Ifitemiediiite  Builder 

12 

...  I'm  contemplating  whether  it  would  be  worth  [it]  to 
demonstrate  it  to  begin  with  or  not. 

Speculator 

13 

Just  to  get  them  to  see  what  it's  doing  before  I  stan  describing  it. 

Advisor  (2) 

14 

And  that,  I  haven't  decided  about. 

Speculator 

15 

But  ru  continue  with  the  overview  idea. 

Facilitator  (1) 

16 

So,  I'd  give  them  the  general  introduction  about  what  it  does  and 
why  it's  so  special. 

Intermediate  Builder 

17 

and  of  course  the  thing  [VST]  might  be  there.  Fd  be  showing 
them  the  thing. 

Intermediate  Builder 
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GOAL  STRUCTURE  FOR  INSTRUCTIONAL  DESIGN 

Previous  sections  have  described  the  actors  and  the  roles  they  play.  In  this  section,  we 
use  the  model  to  look  at  the  structure  of  the  goals  that  guide  the  design  process,  and  compare  the 
procedures  of  two  very  different  designers.  Based  on  analyses  of  their  protocols,  we  identify  two 
styles  of  design:  design  by  plan  and  ^sign  by  feature.  Then  we  use  this  analysis  to  make  some 
general  statements  alimut  the  goal  structure  for  instructional  design. 

Table  4  below  presents  an  excerpt  from  the  protocol  of  a  very  organized  designer  whom 
we  call  Sid.  Sid  was  a  graduate  of  the  teacher  training  progra^  and  so  had  some  experience 
teaching  in  a  high  school  setting.  In  this  excerpt,  Sid  was  ^ginning  a  design  to  teach  energy 
principles  using  the  VST.  (In  the  table,  senior  managers  are  listed  in  bold-face,  middle  managers 
in  plain  typeface,  and  builders  in  italics;  action  numbers  in  the  third  column  refer  to  procedures 
from  Table  1.) 


TABLE 4 

Actors  Participating  in  the  First  Part  of  Sid’s  Design 


Statement _ Actor  (Action  #) 

1  All  right,  this  program  is  exactly  the  material  I’d  be  able  to  use?  Architect  (3) 

This  is  the  final  form  of  the  program? 

2  All  right.  It  looks  like  you’ve  got  all  different  kinds  of  options  Speculator 

because  of  the  different  types  of  energy  sources;  there’s  solar, 

there's  vegetable  matter  so  you  can  bring  in  biology,  there’s 
nuclear  if  you  want  to  bring  in  fission  or  fusion.  You  can  really 
expand.  There  are  a  lot  of  options  there.  And  there’s  a  whole 
different  level  of  the  machine  itself,  which  are  the  principles  that 
you’ve  given  me  here. 

3  So  it  looks  like  this  [principles  card]  would  be  the  place  to  start;  Abstract  Builder 

this  would  be  the  core  of  the  lesson,  with  the  tangents  being  little 

sub-topics  that  might  lead  into  other  units.  By  going  back  to  this 
central  theme  in  class,  the  principles  card,  by  going  back  to  that, 
you  can  do  a  sub-unit  on,  say,  nuclear,  or  a  sub-unit  on  solar,  a 
sub-unit  on  vegetable-type  power. 

4  Okay,  so.  I’m  designing  an  entire  unit,  sort  of  outlining?  Architect  (1) 

5  Fm  thinking  the  best  place  to  stan  would  be  to  follow  the  format  Abstract  Builder 

of  the  review  box 

6  and  start  out  with  familiarizing  the  students —  Intermediate  Builder 

7  Now  I'm  just  starting  a  little  outline  on  my  pad  Secretary 

8  familiarizing  the  students  with  the  parts  of  the  machine,  showing  Intermediate  Builder 

each  pan  separately  and  telling  about  it 
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9  I  would  say  one  day’s  class  for  a  general  science  course 

10  Maybe  the  first  day  would  be  introducing  the  machine  itself, 
talking  a  little  bit  about  the  different . . .  parts  it  has,  and  not 
really- getting  into  any  of  the  specifics  of  how  the  connections 
work. 

11  So  first  Ill  put  "L  introduce  machine,  describe  parts" 

12  I  think  one  class  period  should  be  fine  for  that 

13  Then, . . .  once  the  kids  are  familiar  with  the  parts,  we  could  go 
into  a  little  unit  on  energy  sources ...  I  think  that’s  where  I 
would  go  next 

14  ...  and  on  my  oudine.  I’ll  just  call  it  ’’VST2000  Unit" 

15  and  Ill  put  what  I  guess  would  be  the  ^ount  of  time  I  would 
spend  on  each  topic  in  parentheses. 

16  Secondly  is  sources 

17  ...  Roman  numeral  one,  I  wrote,  "Introduce  machine,  describe 
parts."  and  then  one  day  for  that.  And  Roman  numeral  H.,  I 
just  wrote  "Sources  of  energy  " 

18  and  A.,  I’ll  put  "Solar,"  B.  Nuclear,  C.  Vegetable,  and 
D.  Other,  which  might  not  be  included  in  the  machine 

19  but  I’d  rather  be  complete  and  just  use  the  machine  to  introduce 
as  many  topics  as  possible 

20  Now  it  looks  perfect — one  day  for  each  of  these — because  that's 
one  week 

21  Spend  a  day  for  each  topic 

22  which,  obviously  we're  just  going  to  introduce,  not  go  into  any 
depth 

23  which  is  probably  the  best  thing  for  this  level  anyway 

24  And  I  think  it  would  be  good  at  this  point ...  to  assign  a  long¬ 
term  . . .  unit  project  to  ^e  class. 

25  So,  I'm  going  to  put,  after  Roman  numeral  II.  "Sources  of 
energy"  I'm  going  to  put  "assign  research  project" 

26  Because  already,  since  we’re  only  going  into  limited  depth,  any 
of  these  topics  lends  itself  to  loolang  at  it  in  more  depth 


Intermediate  Builder 
Intermediate  Builder 

Secretary 
Advisor  (3) 
Intermediate  Builder 

Secretary 
Monitor  (1) 

Secretary 

Intermediate  Builder 

Intermediate  Builder 

Advisor  (1) 

Advisor  (3) 

Intermediate  Builder 
Intermediate  Builder 

Advisor(l) 
Intermediate  Builder 

Secretary 


Advisor  (4) 
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The  goals  that  Sid  addressed  in  the  above  section  of  protocol  are  listed  below  in  Table  5 
(statement  numbers  from  Table  4  are  shown  in  parentheses).  Table  5  indicates  that  Sid  used  a 
clear,  well-organized  approach.  He  began  by  clarifying  his  task  and  resources,  by  considering  his 
design  options,  and  by  outlining  a  structure  for  his  overall  design.  When  he  began  the  design 
itself^,  he  worked  in  blocks,  dealing  with  each  block  in  a  similar  fashion.  He  first  decided  to  begin 
a  section.  Then,  within  each  section,  he  accomplished  three  things:  he  determined  how  long  to 
spend  on  each  part;  he  fleshed  out  the  content;  and  he  checked  the  adequacy  of  his  decisions. 


TABLES 

Goal  Structure  for  Initial  Section  of  Sid's  Design 


Clarify  task  constraints  (1) 

Consider  design  options  (2) 

Decide  general  plan  for  design  (3) 

Clarify  task  procedures  (4) 

Decide  start  point  (5-8) 

Determine  timing  for  this  section  (9) 

Flesh  out  topic  (10-11) 

Check  adequacy  of  decisions  so  far/  for  this  section  (12) 
Decide  next  topic/section  (13-19) 

Determine  timing  for  this  section  (21) 

Flesh  out  topic  (22-23) 

Check  adequacy  of  decisions  for  this  section  (20) 
Decide  next  topic/section  (24-26) 


Sid's  approach  in  this  section  of  the  protocol  might  be  described  as  design  by  plan.  He  set 
out  a  general  plan  for  his  design  (statement  3)  before  filling  in  the  specifics,  and  when  he  did  turn 
to  specific  units,  he  worked  on  them  with  an  organized  approach.  We  will  see  later,  in  the  section 
on  influences  of  external  knowledge  on  design,  that  Sid  made  much  use  of  outside  knowledge 
structures  provided  by  the  computer  tutorial  and  task  instructions  to  organize  his  design. 

In  the  goal  structure  outlined  above,  one  can  also  see  a  hierarchical  order  of  logic  in  Sid's 
decisions.  He  obtained  needed  information  about  the  task  before  beginning  it;  he  determined  an 
overall  structure  for  the  design  before  filling  in  the  details;  he  decided  on  a  general  topic  before 
deciding  how  long  a  presentation  would  be;  and  he  decided  how  much  time  to  spend  on  a  unit 
before  determining  what  specific  information  to  present  within  that  time  frame.  He  seemed  to 
check  the  adequacy  of  his  decisions  as  he  made  them.  This,  of  course,  is  not  the  only  logical  order 
that  a  designer  could  choose. 

The  goal  structure  reveals  how  a  particular  designer  approached  the  task.  Sid  was  one  of 
the  more  experienced  designers  in  our  subsample,  had  fairly  good  knowledge  of  the  subject  matter, 
and  was  willing  to  use  existing  information  to  aid  in  his  design.  These  characteristics  may  have 
contributed  to  his  organized  approach.  All  designers  will  not  necessarily  be  as  well-organized. 
Sally  was  a  case  in  point  Recall  that  she  was  a  novice  designer  and  that  she  had  some  difficulty 
remembering  details  of  the  "VST  subject  maner  (see  statements  29  and  32  in  Appendix  1).  Table  6 
below  presents  a  list  of  goals  for  Sally  's  protocol.  (Numbers  in  parentheses  refer  to  statement 
numbers  from  Appendix  1.) 
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TABLE  6 

Goal  Structure  for  First  Half  of  Sally's  Design 


Break  down  information  into  teachable  units  (1) 

Clarify  task  constraints  (2-3) 

Qarify  task  procedures  (4-8) 

Decide  stan  point  (9-10) 

flesh  out  topics  (1 1, 15-16) 

flesh  out  presentation  method  (12-14, 17) 

Consider  content  to  include  (18) 

decide  whether  to  include  content  (19) 

Flesh  out  topic  (19-21) 

Add  more  content  (22-24) 

revise  sequence  (22-23) 

Add  pedagogical  feature  (26) 

flesh  out  method  and  content  (28-38) 

get  missing  information  about  content  (29-30,  32-33) 

Add  more  content  (39-41) 
decide  sequence  (40) 
flesh  out  detail  (43-47) 

Consider  additional  content  (48-50) 
revise  sequence  (48) 

Add  more  content  (51-55) 
decide  sequence  (53) 
flesh  out  presentation  method  (57, 59) 

determine  constraints  on  task  (56,  58) 

Determine  goals  of  instruction  (61-70) 

consider  different  types  of  presentations  for  different  goals  (64.  67) 


In  contrast  to  Sid's  design  by  plan,  Sally's  approach  might  be  described  as  design  by 
feature.  Although  she  began  her  design  similarly  to  Sid  (and  all  the  other  designers),  by  obtaining 
information  necessary  to  construct  the  design,  she  did  not  proceed  by  laying  out  a  plan  and 
progressing  through  the  design  in  discrete  units.  Rather,  she  suggested  items  of  information  or 
activity  as  they  occurred  to  her,  and  her  subsequent  design  consisted  of  a  list  of  topics,  activities 
and  pe^gogical  features,  which  she  attempted  to  fit  into  a  sensible  whole.  This  approach  to 
design  is  si^ar  to  the  opponunistic  planning  described  by  Hayes-Roth  and  Hayes-Roth  (1978). 

Sally's  strategy  led  to  patching  of  her  design — going  back  and  inserting  items  (see 
statements  12, 22-23,  and  48— which  left  the  structure  and  feasibility  of  her  design  unclear.  For 
example,  it  was  not  clear  what  would  be  included  in  the  introduction  versus  later  parts  of  the 
teaching,  when  she  was  going  to  tell  the  students  about  the  differences  between  the  two  types  of 
wires  (statements  18-21),  or  how  many  days  each  item  would  cover.  Ultimately,  Sally  ran  into 
trouble  when  she  realized  that  she  did  not  know  what  the  major  goals  of  her  instruction  were.  This 
lack  of  clarity  in  Sally's  design  suggests  that  the  teacher  implementing  it  would  have  a  great 
influence  on  the  instruction's  ultimate  form  and  outcome. 

From  the  above  analysis  of  the  goals  the  designers  addressed,  we  see  that  the  goal  structure 
(i.e.,  the  goals  set  and  their  sequence)  is  knowledge-driven,  but  can  be  opponunistic,  as  well.  For 
both  designers,  their  decisions  are  influenced  by  the  designer's  knowledge  of  the  task 
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environment,  their  ideas  about  the  components  of  instruction,  and  their  personal  stylistic 
preferences  for  organization  or  opportunity. 

Designers  need  certain  information  about  the  task  environment  before  they  can  create  a 
design.  We  suppose  that  this  statement  is  tme  for  all  types  of  design,  and  that  the  specific 
information  nee^  will  depend  on  the  particular  task.  Some  of  the  information  that  our  designers 
needed  are  an  understanding  of  the  task  requirements,  of  the  content  of  instruction,  of  the  gods  of 
instruction,  and  of  the  resources  available.  Sid  and  Sally  both  showed  an  understanding  of  this 
principle  when  they  clarified  the  task  constraints  and  procedures  as  they  began  their  designs. 

When  designers  lack  any  of  this  information,  they  will  hit  a  stumbling  point  in  their  design  and 
have  to  obtain  it,  as  indicated  by  Sally’s  statements  numbered  29, 32, 56,58,  and  61-70. 

The  goals  that  Sid  and  Sally  addressed  reveal  the  components  that  they  thought  ought  to  be 
included  in  ^eir  instructional  designs.  These  conqronents  are  content  and  activities  (both  general 
and  specific),  order  of  presentation,  timing,  and  presentation  methods  (including  materials). 
Different  designers  may  emphasize  different  components.  Sid  included  more  of  the  listed 
components,  and  included  them  more  consistently,  than  Sally  did. 

Designers  also  possess  their  own  styles  for  addressing  the  task,  which  may  depend  on  their 
level  of  experience  with  design,  knowledge  of  the  content,  and  personal  preferences  for  how  to 
construct  a  design.  We  have  seen  two  styles:  d«/g/i  by  p/fl«  and  design  by  features.  Analysis  of 
additional  protocol  data  would  be  likely  to  reveal  others.  The  point  is  that  a  designer’s  preferred 
style  will  influence  the  goals  he  or  she  addresses,  and  the  order  in  which  he  or  she  addresses  them. 


Knowledge  Used  in  Instructional  Design 

This  section  examines  types  of  knowledge  that  were  used  in  relation  to  the  functional  roles 
identified  in  the  model,  and  the  role  of  knowledge  in  creating  the  design. 

Each  actor  needs  certain  types  of  knowledge  in  order  to  perform  its  function.  Whereas  the 
functions  that  the  actors  serve  are  general  and  can  apply  to  a  variety  of  design  tasks  (see  Table  1), 
the  knowledge  the  actors  use  is  specific  to  the  type  of  design  being  created— in  this  case, 
instrucdonalxiesign.  This  section  describes  some  types  of  knowl^ge  that  each  actor  might  use  in 
designing  instruction,  and  discusses  the  sources  of  that  knowledge.  Then,  it  proposes  a  basis  on 
which  the  pedagogical  content  knowledge  contained  in  the  design  is  generated.  In  a  later  section, 
we  will  analyze  specific  influences  of  knowledge  on  design  activity. 

TYPES  OF  KNOWLEDGE  USED  BY  ACTORS 

Table  7  lisis  some  types  of  knowledge  that  the  actors  would  use  in  creating  a  design  for 
instruction.  This  list  characterizes  the  range  and  main  types  of  information  tiiat  different  actors 
would  use,  but  is  not  meant  to  be  exhaustive.  The  types  of  information  were  determined  by 
analyzing  the  role  of  each  actor  and  its  relationship  to  the  knowledge  types  that  we  identified  in 
analyzing  the  16  design  protocols  that  we  obtained  (see  Figure  3). 
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TABLE? 

Knowledge  Used  by  Design  Committee  Members 


Problem-space  Construction 


Monitor  •  current  design  process 

•  model  of  instructional  design  process 

Memory  •  content,  i.e.,  VST  &  science 

•  personal  experiences  and  characterisdcs  as  teacher  and  learner 

•  designer's  long  term  memory 

Architect  •  design  task  requirements,  including  goals  and  constraints 

•  teaching  context,  including  available  resources  &  students 


Design  Construction 


Facilitator  •  model  of  instructional  components 

•  design  task  requirements 

•  current  state  of  design 

Advisor  •  general  pedagogical  heuristics 

•  pedagogical  content 

•  teaching  context,  including  available  resources  &  students 

•  model  of  good  instruction 

•  current  state  of  design 

Speculator  &  Builders 

•  model  of  instructional  com]X)nent5 

•  content,  i.e.,  VST  &  science 

•  pedagogical  content 

•  teacldng  context,  including  available  resources  &  students 

•  current  state  of  design 

Secretary  •  design  decisions  made  by  builders 


The  monitor,  who  keeps  track  of  the  design  process,  employs  information  about  that 
process,  including  knowledge  of  the  procedures  as  they  take  place  and  a  model  of  the  course  that 
the  process  ought  to  follow — i.e.,  an  instructional  design  schema.  The  memory  may  retrieve  any 
information  from  the  designer's  long-term  store,  but  is  especially  likely  to  employ  information 
concerning  the  instructional  content  and  the  designer's  personal  experiences  as  a  teacher  or  learner. 
The  architect,  who  is  the  primary  builder  of  the  space  in  which  the  design  is  constructed,  has 
knowledge  of  constraints  on  the  task,  including  task  requirements  and  the  setting  in  which  the 
instruction  will  take  place.  The  facilitator,  who  coordinates  the  design  construction,  employs 
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information  about  the  requirements  of  the  task,  the  current  state  of  the  design,  and  a  model  of  the 
components  that  instruction  ought  to  include — i.e.,  an  insmictional  schema  (compare  instructional 
design  schema,  described  above).  The  advisor,  who  provides  pedagogical  ^vice  and  evaluates 
the  design,  possesses  knowledge  of  guidelines  for  teaching  in  gene^  (pedagogical  heuristics)  and 
for  teaching  this  material  in  particular  (pedagogical  content),  beliefs  about  the  setting  in  which 
instruction  will  occur,  an  evaluative  model  of  what  constitutes  "good"  instruction,  and  knowledge 
of  the  current  state  of  the  design.  The  speculator  and  builders  perform  similar  functions  in  creating 
the  design  and  use  the  same  types  of  knowledge,  i.e.,  a  model  of  components  to  be  included  in 
instruction  (instructional  schema),  knowledge  of  the  content  being  taught  and  how  to  teach  it, 
information  about  the  setting  in  which  instruction  will  occur,  and  the  current  state  of  the  design. 
The  secretary  records  decisions  made  by  the  builders,  and  so  needs  only  information  about  those 
decisions. 

The  examples  below  illustrate  most  of  the  types  of  knowledge  listed  in  Table  7.  The  reader 
can  find  the  context  for  these  examples  in  the  protocols  in  Table  4  and  Appendix  L  Note  that  the 
designers  may  use  several  types  of  knowledge  in  any  given  statement,  and  the  following  examples 
often  contain  more  than  one  type.  Also  note  that,  in  practice,  knowledge  may  be  either  explicitly 
stated  or  implied.  Examples  of  both  types  appear  below,  with  explanations  provided  for  examples 
in  which  the  knowledge  is  implied. 

Model  of  design  process:  "I  have  to  decide  what  my  focus  is"  (Appendix  I,  statement  63) 

Personal  experience:  "I'm  used  to  doing  a  presentation  after  I've  thought  about  it," 
(Appendix  I,  statement  7).  "Basically  [in  teaching,  I  would]  sort  of  follow  how  I  learned  to  do  it." 
(Appendix  I,  statement  10;  indicates  an  implicit  understanding  of  how  the  designer  learned). 

VST2000:  "like  a  car,  it  needs  a  source  of  its  own  power  to  get  going”  (Appendix  1. 
statement  19).  "Give  them  a  table  and  have  it  set  up  for  them.  One  column  for  each  unit,  what 
kind  of  energy  it  produces,  what  it  needs-source  of  raw  energy."  (Appendix  I,  statement  28; 
indicates  implicit  understanding  of  VST  units  and  energy  requirements.) 

Science:  "(Yjou've  got  all  different  kinds  of  options  because  of  the  different  energy 
sources.  There's  solar;  there's  vegetable  matter,  so  you  can  bring  in  biology;  there's  nuclear  if  you 
want  to  bring  in  fission  or  fusion"  (Table  4,  statement  2). 

Design  Task:  "Are  they  going  to  be  using  the  computer  at  all,  or  are  they  going  to  be 
learning  to  operate  this?"  (Appendix  I,  statement  2).  "Then  I  just  orally  tell  you  how  I'd  set  this" 
(Appendix  I,  statement  4).  "How  would  I  break  it  down  to  teach  something  like  that?"  (Appendix 
I,  statement  1;  indicates  implicit  understanding  that  the  task  is  to  teach). 

Context-available  resources:  "I  don't  know  whether  I'd  have  30  computers"  (Appendix  I, 
statement  56). 

Context- students:  "[This  design]  is  probably  the  best  thing  for  this  level  [of  student] 
anyway"  (Table  4,  statement  23). 

Instructional  components:  "And  I  think  it  would  be  good  at  this  point . .  .to  assign  a  long¬ 
term  . . .  unit  project  to  the  class"  (Table  4,  statement  24). 

Pedagogical  heuristics:  "I'd  rather  be  complete,  and  just  use  the  machine  to  introduce  as 
many  topics  as  possible"  (Table  4,  statement  19).  "It'd  be  good  to  give  each  kid  a  feel,  to  try  to 
move  the  switch"  (Appendix  I,  statement  57).  "I  think  what  I  would  do  first  is  give  them  a  general 
overview  of  what  it  is"  (Appendix  I.  statement  9;  indicates  implicit  belief  that  instruction  should 
begin  with  general  information). 
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Pedagogical  content:  "I'm  thinking  the  best  place  to  start  would  be  to  follow  the  format  of 
the  review  box"  (Table  4,  statement  5). 

Model  of  good  design:  "Now  it  looks  perfect-one  day  for  each  of  these-because  that's 
one  week."  (Table  4,  statement  20). 

SOURCES  OF  KNOWLEDGE 

We  distinguish  two  major  sources  of  knowledge — internal  and  external.  Internal 
information  is  retrieved  from  the  designer’s  memory.  External  information  is  obtained  from 
sources  outside  the  designer’s  mind,  such  as  notes,  diagrams,  the  computer  tutorial,  or  the 
experimenter.  Internal  knowledge  consists  of  two  types  of  information:  pre-existing  and 
generated.  Pre-existing  knowledge  is  the  information  and  beliefs  stored  in  memory  that  an 
individual  brings  to  the  task.  It  includes  such  things  as  knowledge  of  content,  pedagogy,  and 
personal  experiences.  Generated  knowledge  is  ad  hoc  information,  created  on  the  spot  to  meet  a 
particular  need.  Some  examples  of  generated  knowledge  are  information  about  the  state  of  the 
design,  the  current  design  process,  and  designer-supplied  goals  and  constraints.  A  panicularly 
interesting  example  of  generated  knowledge  in  these  daw  is  pedagogical  content  knowledge.  The 
next  section  explores  the  generation  of  pedagogical  content  knowledge  in  these  data.  In  a  later 
section,  we  will  look  at  the  influences  of  various  types  of  knowledge  on  instructional  design. 

GENERATED  PEDAGOGICAL  CONTENT  KNOWLEDGE 

Pedagogical  content  knowledge  consists  of  information  about  how  to  teach  the  panicular 
content  under  consideration.  It  is  the  type  of  knowledge  represented  in  the  design  artifact  itself.  In 
the  present  data,  pedagogical  content  knowledge  is  primarily  generated  during  the  design  proces^. 
Note  that  pedagogical  content  knowledge  does  not  necessarily  have  to  be  generated  during  design; 
experienced  designers  could  possess  pedagogical  content  knowledge  as  part  of  their  expertise,  for 
example,  in  how  to  teach  history  or  math.  However,  this  study  provides  an  unusual  opportunity  to 
observe  pedagogical  content  knowledge  being  generated,  because  it  was  structured  so  that  the 
participants  were  designing  instruction  about  content  that  they  had  never  taught  before. 

This  section  focuses  on  how  pre-existing  knowledge  is  used  in  generating  pedagogical 
content  knowledge.  In  our  view,  pedagogical  content  knowledge  is  created  using  pedagogical 
knowledge,  content  knowledge,  personal  experiences  related  to  the  content,  and  knowledge  of  the 
teaching  context.  The  process  by  which  it  is  generated  might  be  considered  analogous  to  carrying 
on  a  conversation.  In  that  case,  the  artifact  (that  is,  the  conversation)  is  created  as  the  pardes 
speak,  and  the  particular  form  it  takes  derives  from  the  speakers’  implicit  understanding  of  how  to 
communicate,  their  knowledge  of  the  subject  matter  being  discussed,  their  personal  feelings  and 
experiences  related  to  that  subject  matter,  and  the  context  of  the  conversation.  In  the  same  way,  the 
design  artifact  is  created  using  an  implicit  understanding  of  how  to  teach,  knowledge  of  and 
experience  learning  the  subject  matter  to  be  taught,  and  the  context  in  which  the  teaching  will 
occur. 


We  will  use  Sid's  protocol  to  explore  the  relation  between  existing  and  generated 
knowledge.  As  indicated  previously  (see  Table  7),  the  builders,  speculator  and  advisor  are  the 
actors  most  likely  to  use  pedagogic^  content  knowledge.  Looking  at  the  section  of  Sid's  protocol 
presented  in  Table  4,  one  can  see  that,  indeed,  the  statements  made  by  these  actors  were  the  ones 
that  contained  knowledge  about  how  to  teach  this  particular  material.  Many  pieces  of  knowledge 
contributed  to  the  pedagogical  content  statements  in  Sid's  protocol.  Table  8  below  suggests  some 
specific  beliefs  and  information  that  seem  to  have  played  a  role.  Statement  numbers  from  Sid's 
protocol  in  Table  4  and  knowledge-types  corresponding  to  each  belief  are  indicated. 
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TABLES 

Specific  Information  Contributing  to  Sid's  Pedagogical  Content  Knowledge, 
with  Knowledge  Types  and  Statement  Numbers  (from  Table  4) 


K-Tvpe 

Information  and  Beliefs 

2,3 

content 

types  of  energy  sources,  VST  machine,  and  scientific  principles; 

3 

pedagogy 

tegin  with  an  overall  structure  for  the  instruction; 

5 

learning  exp 

the  tutorial  review  box  presents  these  topics  in  a  sensible  order  for 
teaching; 

5 

ped  content 

II 

6,  8 

learning  exp 

the  tutorial  begins  by  familiarizing  the  learner  with  the  parts  of  the 
machine 

9 

pedagogy 

knowledge  of  how  much  material  can  be  covered  in  one  day 

_9 

teach  context 

how  much  material  can  be  covered  in  this  type  of  classroom  with 
this  type  of  student; 

10 

pedagogy 

in  the  beginning,  give  a  general  presentation;  don't  get  too 
specific; 

10 

content 

VST  parts  and  connections; 

13 

pedagogy 

students  should  learn  one  lesson  before  beginning  a  new  one 

17,  18 

pedagogy 

information  should  be  pre*— *ed  hierarchically ; 

18 

content 

three  energy  sourcer  a^e  used  in  the  VST; 

18 

content 

other  energy  sources  are  used  in  the  world 

19 

pedagogy 

be  complete 

19 

pedagogy 

use  available  resources  to  introduce  as  many  topics  as  possible 

20,  21 

pedagogy 

it  is  good  to  -ever  one  tonic  per  day 

20 

teach  context 

an  academic  week  consists  of  five  days 

20 

pedagogy 

it  is  good  if  lessons  fit  into  a  five-day/one-week  package; 

22 

pedagogy 

in  one  day,  you  cannot  get  into  too  much  depth  on  a  topic 

23 

teach  context 

it  is  best  to  begin  by  giving  this  level  of  student  only  an 
introduction  to  the  material; 

23 

pedagogy 

II 

24 

pedagogy 

it  is  good  to  have  a  variety  of  activities; 

24 

pedagogy 

it  is  good  to  make  students  responsible  for  doing  a  project  on  their 
own; 

24,26 

pedagogy 

it  is  good  for  students  to  pursue  their  particular  interests  in  more 
depth; 

26 

teach  context 

class  time  doesn't  allow  for  looking  at  all  valuable  topics  in 
sufficient  depth 

The  knowledge  types  represented  in  Table  8  span  the  four  types  frem  which  we  claim 
pedagogical  content  knowledge  is  derived,  that  is,  knowledge  of  p^gogy,  content,  teaching 
context  and  learning  experience.  Note  that  the  third  piece  of  information  listed  in  Table  8 — that  the 
tutorial's  review  box  presents  information  in  an  sensible  order  for  teaching — is  identified  as 
pedagogical  content  toowledge  (as  well  as  learning  experience).  Even  though  we  cannot  identify 
the  specific  pieces  of  infoimation  on  which  this  information  is  based,  it  is  likely  that  the  knowledge 
was  generate  on  the  spot  because  the  designers  were  not  told  prior  to  being  presented  with  the 
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task  that  they  would  be  designing  instruction.  This  example  seems  to  indicate  the  spontaneity  with 
which  pedagogical  content  Imowledge  can  be  generated. 

Next,  we  want  to  show  how  the  above  knowledge  is  related  to  the  artifact.  The  following 
analysis  describes  how  each  piece  of  knowledge  listed  in  Table  8  above  relates  to  the  pedagogical 
content  knowledge  generated  in  the  excerpted  section  of  protocol. 

Sid  began  his  design  by  listing  different  options  for  what  to  present  in  his  instruction 
(statement  2).  This  required  knowledge  of  the  content:  energy  sources,  scientific  principles  and  the 
machine  itself.  Next  he  chose  a  genei^  smicmrc  for  the  content,  which  we  assume  was  based  on  a 
pedagogical  belief  that  it  is  good  to  begin  with  an  overall  structure.  Statement  5  reflected  a  decision 
to  follow  the  sequence  of  topics  presented  in  the  review  box  of  the  tutorial.  We  infer  that  this 
decision  was  based  on  the  belief  that  the  review  box  presents  information  in  a  sensible  order  for 
teaching.  The  next  step  (statements  6  and  8)  was  to  flesh  out  that  decision,  explaining  the  first 
topic  and  how  it  would  be  presented.  This  decision  was  likely  based  on  the  inJFormation  presented 
in  the  tutorial.  The  subsequent  decision  (statement  9)  concerned  how  much  material  can  be 
covered  in  one  day  (with  the  implicit  limitations  of  the  teaching  context:  this  type  of  classroom  and 
this  type  of  student).  This  was  followed  by  a  further  fleshing  out  of  the  first  day's  presentation 
(statement  10),  bas^  on  the  apparent  belief  that,  in  the  beginning,  it  is  best  to  give  a  general 
presentation  and  not  get  too  specific. 

Sid  then  made  an  implicit  judgment  that  the  first  topic  had  been  covered  sufficiently,  and 
decided  to  go  on  to  the  next  topic  (statement  13).  In  deciding  that  the  next  unit  will  be  on  particular 
energy  sources  (statements  17  and  18),  he  departed  from  his  original  plan  to  follow  the  sequence 
of  topics  in  the  review  box  (energy  sources  are  not  listed  in  the  review  box),  and  determined  what 
information  he  thought  it  was  best  to  teach  next.  The  que.stion  arises;  where  did  this  decision  come 
from?  As  will  be  described  later,  these  topics  are  near  the  top  of  a  hierai’chical  undersunding  of  the 
VST.  We  infer  that  the  decision  to  cover  the  energy  sources  next  was  based  on  the  designer's 
understanding  of  the  VST  combined  with  the  implicit  pedagogical  belief  that  it  is  best  to  teach 
according  to  a  hierarchical  knowledge  structure. 

Sid  knew  that  lliree  types  of  energy  sources  are  included  in  the  VST,  and  decided  to  spend 
a  day  on  each  one,  with  an  extra  day  for  energy  sources  that  are  not  included  in  the  machine 
(statements  18  and  21).  simultaneously  assessing  that  his  design  was  good  (statement  20).  We 
infer  that  these  decisions  were  based  on  an  understanding  of  the  school  calendar  and  the  teliefs  that 
it  is  best  to  cover  one  topic  per  day  and  it  is  good  if  the  lessons  fit  into  a  neat  one-week  package. 
Sid  again  used  his  judgment  concerning  how  much  material  can  be  covered  in  one  day,  and 
reinforced  this  decision  with  the  belief  that  this  level  of  student  only  needs  an  introduction  to  the 
material  (statements  22  and  23). 

Finally,  Sid  decided  to  insert  a  project  assignment  in  which  the  students  would  go  into  one 
topic  in  more  depth  (statements  24  and  26).  The  beliefs  leading  to  this  decision  might  stated  in  a 
variety  of  ways:  it  is  good  to  have  variety  of  activities;  it  is  good  to  make  students  responsible  for 
doing  a  project  on  their  own;  it  is  good  for  students  to  pursue  their  particular  interests  in  more 
depth;  and  class  time  doesn't  allow  looking  at  all  valuable  topics  in  depth. 

In  sum,  this  analysis  shows  that  four  types  of  pre-existing  knowledge  can  be  used  to 
generate  pedagogical  content  knowledge:  knowledge  of  pedagogy,  content,  teaching  context  and 
learning  experience.  It  also  supports  the  claim  that  the  builders,  speculator  and  advisor  are  the 
actors  most  likely  to  use  pedagogical  content  knowledge. 
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Influences  of  Knowledge  on  Instructional  Design 

The  knowledge  a  designer  possesses  influences  the  goals  that  arc  set  and  how  those  goals 
arc  accomplished.  This  section  discusses  influences  of  pre-existing,  generated,  and  external 
knowledge  on  both  the  design  process  and  design  artifact 

INFLUENCES  OF  PRE-EXISTING  KNOWLEDGE 

General  considerations  suggest  that  certain  pieces  of  pre-existing  knowledge  influence  the 
outcome  of  design.  For  example,  the  designer's  model  of  instructional  components  determines 
what  the  designer  will  include;  the  designer's  model  of  good  design  sets  standards  for  how  the 
design  will  be  evaluated;  and  pedagogical  heuristics  ^de  how  the  designer  will  stmeture  the 
content,  presentation,  sequencing,  and  timing  of  the  instruction.  In  fact  we  infer  the  content  of 
these  types  of  information  brom  the  designer's  activities.  For  example,  we  identify  the  pedagogical 
heuristics  a  designer  used  by  analyzing  £e  protocol  data. 

To  address  some  questions  about  the  role  of  pre-existing  knowledge  in  design  without 
depending  on  inferences  based  on  design  protocols,  we  conducted  an  analysis  of  VST2000 
knowledge  in  design.  VST  information  was  used  extensively  in  the  design  task,  and  we  could 
derive  an  elaborated  description  of  it  from  the  tutorial  materials,  without  making  inferences  from 
the  protocol  data.  We  assumed  a  semantic  network  structure  to  the  information  and  developed  a 
computer  model  that  represents  VST2000  knowledge. 

SEMANTIC  NETWORK  REPRESENTATION  OF  THE  VST2000 

The  semantic  network  representation  of  the  VST  2000  knowledge  was  constructed  by  a 
computer  program  from  a  list  of  propositions.  These  propositions  were  obtained  through  an 
analysis  of  the  tutorial  materials.  Each  proposition  consists  of  a  node-rclation-node  triplet  and  a 
direction  indicator  that  provides  information  about  how  to  order  the  nodes  within  the  proposition. 
The  resulting  semantic  network  does  not  retain  the  structure  of  the  tutorial  from  which  the 
designers  learned  about  the  VST — that  is,  it  does  not  contain  the  same  order  of  topics,  exercise 
questions,  or  structure  of  the  descriptive  text  Rather,  it  is  a  hypothetical  representation  of  the 
information  that  designers  could  have  obtained  from  the  tutorial  instruction.  A  portion  of  the 
semantic  network  generated  by  our  computer  program  is  represented  diagramatically  in  Figure  6. 
This  selection  represents  about  10%  of  Ae  totd  network  generated. 

One  can  calculate  the  distance  between  any  two  nodes  in  the  network  by  counting  the 
number  of  relational  links  that  must  be  followed  to  connect  the  two  nodes.  The  distance  from  the 
root  node  (in  this  case, VST 2000 )  to  a  particular  node  indicates  the  levei  of  that  node  in  the 
netwoik.  As  Figure  6  indicates,  ^e  network  has  a  hierarchical  structure;  nodes  lower  in  the 
network  are  components  or  descriptions  of  higher-level  nodes. 

RELATIONSHIP  OF  SEMANTIC  NETWORK  TO  DESIGN 

An  examination  of  the  protocol  data  in  conjunction  with  the  semantic  netwoik  model  , 
revealed  several  hypotheses  concerning  how  knowledge  in  the  semantic  network  relates  to  the 
design.  These  hypotheses  arc: 

•  High-level  knowledge  (adjacent  to  the  root  node)  is  presented  first  in  instruction; 

•  Information  near  the  most  recently  used  node  is  often  included  next; 

•  Low-level  knowledge  (far  from  the  root  node)  is  often  omitted  from  the  design; 
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•  Similarity  between  sections  of  the  network  is  used  to  advantage  in  the  design; 

•  Incomplete  knowledge  leads  to  a  breakdown  or  refocusing  of  design  activity;  and 

•  Design  process  is  not  necessarily  either  breadth-first  or  depth-first. 

The  first  four  hypotheses  involve  the  structure  of  the  design  artifact  relative  to  the  structure 
of  the  semantic  network.  The  first  three  taken  together  indicate  tlut  information  about  the 
VST2000  in  the  designs  tended  to  reflect  the  semantic  network  stmcture  generated  by  our 
computational  model,  rather  than,  for  example,  the  didactic  structure  of  the  tutorial  or  the  spatial 
structure  of  the  schematic  diagram  of  the  VST.  The  last  two  hypotheses  involve  the  design  process 
relative  to  the  semantic  network.  We  will  describe  each  hypothesis  in  turn,  citing  an  example  for 
each. 


Higher-level  knowledge.  It  is  well-reco^ized  that  higher-order  information  in  a  semantic 
network  is  typically  accessed  first  when  people  recall  information  from  memory  (e.g.,  Kintsch, 
1974).  Stevens  and  Collins  (1977)  found  that  higher-order  information  in  a  semantic  network  was 
typically  presented  first  by  tutors.  Our  data  indicate  that  higher-order  information  also  was 
presented  first  in  instructional  design — that  is,  knowledge  from  nodes  adjacent  to  the  root  node  in 
the  network  were  among  the  first  to  be  presented.  For  example,  one  of  the  first  things  Sid  did  was 
to  describe  the  different  types  of  energy  sources  ot  the  VST2(X)0  (Table  4,  statement  2).  The 
nodes  describing  these  energy  sources  (solar,  vegetable  matter,  and  nuclear  energy  sources)  arc 
directly  linked  to  the  root  VST  2000  node  in  Figure  6. 

Adjacent  knowledge.  Knowledge  from  nodes  near  the  most  recently  used  node  were  often 
included  in  the  design  next.  The  structure  of  the  semantic  network  requires  that  adjacent  nodes 
contain  related  knowledge.  In  fact,  many  branches  of  our  network  contain  groups  of  nodes  and 
relations  that  describe  the  features,  sub-parts,  and  capabilities  of  a  particular  VST  2000  component. 
When  a  designer  included  a  VST  2000  component  in  the  design,  he  or  she  often  include 
information  related  to  that  component  from  adjacent  nodes. 

Lower-level  knowledge.  Knowledge  from  nodes  at  low  levels  in  the  network  is  often  left 
out  of  the  design.  In  some  cases,  knowledge  from  these  lower  level  nodes  was  simply  not 
mentioned,  but  in  other  ca^es  it  was  explicitly  excluded  from  the  design. 

Similar  knowledge.  Designers  sometimes  borrowed  pieces  of  the  design  about  one  pan  of 
the  VST  and  applied  them  to  a  different  pan  if  the  representation  of  the  knowl^ge  for  those  two 
pans  was  similar.  For  example,  Sally  noted  that  the  VST  2000  power  units  are  similar:  “the  units 
arc  basically  the  same ...  I  ^ink  if  you  just  teach  one,  what  you  learn  on  one  you  can  extn^olate 
to  the  other  two  relatively  easily.”  Her  recognition  of  the  sinr^arity  between  the  units  allow^  her 
to  simplify  her  design  task  by  concentrating  on  teaching  the  operation  of  one  unit  in  detail  and 
applying  that  solution  to  similar  parts. 

Incomplete  knowledge.  When  the  designer's  representation  of  the  knowledge  in  the 
network  was  poorly  elaborated  or  incomplete,  a  breakdown  or  refocusing  of  the  design  activity 
occurred.  We  first  identified  this  phenomenon  in  our  analysis  of  the  goal  structure  in  the  previous 
section.  The  principle  is  reiterated  when  we  consider  the  semantic  network  in  relation  to  the 
protocols.  For  example,  Sally  (Appendix  I,  statements  32-34)  was  unable  to  remember  the  names 
of  the  switch  settings.  We  infer  that  her  semantic  network  was  weak  in  this  area.  She  could 
expend  effort  to  remember  these  details,  which  would  lead  to  a  momentary  breakdown  in  design 
activity  as  she  focused  on  obtaining  that  information,  or  she  could  ignore  the  missing  information 
and  continue  her  design  without  it  (as  she  did  in  statement  35). 
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Sequence  of  design  process.  Though  the  design  artifact  itself  tends  to  reflect  the  structure 
of  the  semantic  network,  the  design  process  does  not  necessarily  do  so.  When  designers  included 
knowledge  from  a  particular  node  in  the  network,  they  had  a  variety  of  choices  about  what  to 
discuss  next  They  could  explore  nodes  that  are  subordinate,  superordinate,  on  the  same  level  as, 
or  distant  from  the  node  just  discussed.  The  decision  could  depend  on  what  they  had  covered  so 
far,  the  integrity  of  the  knowledge  in  memory,  how  interesting  or  important  the  information  was  to 
the  design,  and  other  factws.  The  design  process  can  take  a  complicated  path  through  the 
network,  and  does  not  necessarily  follow  either  a  breadth-first  or  depth-^t  expansion. 

To  illustrate  this  situation,  when  Sally  first  discussed  the  VST  2000,  she  described  the 
differences  between  the  two  kinds  of  wires  on  the  machine  (Appendix  I,  statements  18-22).  In  the 
semantic  network,  the  nodes  describing  the  wires  are  actually  two  levels  from  the  root  node,  so  the 
order  in  which  she  discussed  the  VST2000  in  her  design  process  did  not  match  the  semantic 
network  representation.  However,  in  statements  22-23,  Sally  indicated  that  the  instruction  itself 
would  describe  the  different  kinds  of  energy  used  by  the  units  (which  are  closer  to  the  root  node) 
before  it  described  tne  wires.  Therefore,  the  smicture  of  her  design  artifact  more  closely  match^ 
the  structure  of  the  semantic  network  than  the  design  process  did. 

INFLUENCES  OF  GENERATED  KNOWLEDGE 

The  primary  influence  of  generated  knowledge  on  design  can  be  summarized  as:  what  the 
designer  has  already  done  affects  what  he  or  she  will  do  in  the  future;  or  more  simply:  where  one 
goes  depends  pardy  on  where  one  has  been.  Generated  knowledge  of  the  current  design  state  and 
of  the  decisions  made  by  actors  have  an  important  influence  on  the  subsequent  content  that  is 
included  and  tasks  that  are  accomplished.  This  is  what  gives  the  designs  coherence,  and  allows 
one  topic  to  flow  into  another.  Sid's  and  Sally's  protocols  both  demonstrate  this  principle:  "Then. 

. . .  once  the  kids  are  familiar  with  the  pans,  we  could  go  into  a  little  unit  on  energy  sources  ..si 
think  that's  where  I  would  go  next,"  (Table  4,  Statement  13);  and  ". . .  actually,  at  this  point.  I’ve 
gone  through  one,  and  then  I  would  elaborate  on  each  of  these  components . . .  what  the  unit  is 
and  what  it  does  . . .  orient  them  to  each  particular  component  on  the  flow,"  (Appendix  I, 
statements  51-52). 

Some  tasks  cannot  logically  be  completed  until  certain  knowledge  has  been  generated.  For 
example,  one  cannot  evaluate  the  design  until  it  is  complete  enough  to  tear  scrutiny.  Also,  certain 
decisions  naturally  lead  into  other  decisions.  For  example,  the  designer  generates  constraints  for 
the  task  that  guide  the  subsequent  parts  of  the  design.  This  is  demonstrated  in  Sally's  protocol 
when  she  decides  that  it  is  possible  for  each  student  to  have  a  computer,  and  then  adjusts  her 
design  to  include  students  working  individually  on  the  machines  (Appendix  L  statements  56-59). 

INFLUENCES  OF  EXTERNAL  KNOWLEDGE 

The  two  main  functions  of  external  knowledge  are  to  provide  information  the  designer  is 
lacking  (e.g.,  about  the  r^uirements  of  the  design  task  or  features  of  the  VST2(X)0),  and  to 
suggest  ideas  for  the  design  artifact  (e.g.,  about  structure,  content,  or  sequencing  of  the  design). 

In  our  study,  the  external  sources  of  information  were  primarily  the  interviewer,  the  computer 
tutorial,  notes  the  designer  had  made  while  learning  about  the  VST  2000,  and  a  principles  card, 
which  was  presented  as  part  of  the  task  instructions  and  which  listed  energy  principles  that 
designers  could  consider  including  in  their  designs.  The  precise  way  in  which  external  knowledge 
was  used  depended,  in  part,  on  its  form  and  structure.  Below,  we  present  examples  from  the  data 
that  demonstrate  how  knowledge  from  external  sources  influences  design  activity. 


Supply  Missing  Information  about  the  Task  Environment  When  designers  lacked 
necessary  information,  they  often  went  to  external  sources  to  obtain  it  Identifying  and  obtaining 


Model  for  Design  Problem-Solving 
Korpi,  Greeno,  and  Jackson 


page  3 1 


missing  information  is  the  third  function  of  the  Architect,  listed  in  Table  1.  The  types  of 
information  obtained  usually  concerned  the  requirements  of  the  task  and  the  details  about  the  VST 
2000.  When  designers  needed  information  about  task  requirements,  they  turned  to  the  interviewer 
for  clarification.  In  fact,  all  of  the  designers  began  the  task  this  way.  Sid's  and  Sally's  protocols 
illustrate  this  (see  Appendix  I,  statements  2-3  and  Table  4,  statement  1).  When  designers  needed 
information  about  the  VST  2000,  they  might  turn  to  a  variety  of  sources.  In  the  following  excerpt, 
the  designer  used  both  a  diagram  from  the  computer  tutorial  and  the  interviewer  to  clarify  the 
functioning  of  the  VST: 

Can  you  review  [for]  me  again  on  this  . . .  diagram,  where  it  says  the  power — Okay,  it 
means  that  when  this  is  on  ...  it  goes  to  pick  up — well,  for  example,  where  is  this  power 
coming  from?  (Designer  G4B,  speech  5.8) 

Suggest  Ideas  for  the  Design  Anifact .  Designers  used  external  knowledge  in  creating  the 
design  artifact  in  several  ways:  to  structure  their  designs,  to  cue  themselves  about  information  to 
include  in  their  designs,  and  to  incorporate  knowledge  directly  into  their  designs. 

Sid  used  the  principles  card  to  broadly  structure  his  design  (Table  4,  statement  3).  Later, 
he  compared  the  principle's  card  to  his  design  to  determine  what  he  had  covered  and  where  he 
needed  to  go: 

Okay,  now  I'm  looking  back  at  your  principles  card.  We  sort  of  touched  on  storing  energy 
a  little  bit,  because  that  would  come  under  the  lesson  on  potential — not  how  it's  stored,  but 
what  stored  energy  is  and  what  can  be  done  with  it.  Kinetic  sort  of  touches  on  extracting 
stored  energy;  converting  has  been  introduced.  But  we  haven’t  gone  into  transporting,  and 
we  haven’t  gone  into  purifying.  (Sid.  statements  83-84) 

Within  the  overall  structure  defined  by  the  principles  card,  Sid  used  other  external 
information  to  make  local  decisions  about  his  design.  In  the  following  example,  he  used  a  diagram 
of  the  VST  2000  and  the  topic  review  box.  both  of  which  were  pan  of  the  computer  tutorial,  to 
determine  his  next  move: 

I'm  looking  at  the  diagram  now  to  see  where  1  would  go  next.  I’m  looking  at  the  review 
box  ...  to  see  if  there's  a  particular  topic  that  would  make  the  most  sense  to  go  to  next. 

I'm  looking  for  any  kind  of  relation  between  the  energy  sources  and  the  machine.  (Sid, 
statement  30) 

Later,  he  referred  to  the  review  box  to  seek  a  title  for  his  current  topic,  and  the  review  function  of 
the  tutorial  to  suggest  information  that  he  ought  to  include  in  his  instruction: 

I'm  looking  at  the  review  box  to  see  if  there  would  be  a  good  tide  to  call  this.  There  are  a 
lot  of  topics  in  the  review  box:  extracting  energy,  transporting  it,  forms,  converting, 
purifying — what  I'm  going  to  do  is  I’m  going  to  look  at  it  right  now  and  see  if  it's  good, 
and  then  go  to  the  forms  of  energy  [in  the  tutorial  instniction].  The  screen  says  "it  uses 
four  forms  of  energy:  raw,  impulse,  purified,  and  mechanical.  In  addition,  local  electricity 
must  be  supplied."  I  haven't  told  them  about  those  forms  of  energy  yet,  which  I  would 
have  to  do,  maybe.  (Sid,  statements  66-71) 

Sometimes  designers  used  external  information  to  remind  themselves  about  features  of  the 
VST  while  simultaneously  incorporating  that  information  into  the  design.  In  the  following 
example,  Sally  used  her  own  notes  to  refresh  her  memory  about  the  energy  flow  in  the  VST,  while 
proposing  to  use  this  diagram  as  pan  of  the  insmiction; 
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But  I’m  going  to  give  them  this  flow  chart  I  did:  the  unit,  you  get  the  raw  energy,  it 
goes  to  the  converter,  it’s  impulse  energy,  and  then  it  goes  to  I-switches,  it  goes  to  the 
purifier — and  give  them  this  little  flow  chart  to  follow.  And  then  how  it  goes  from  the  I- 
switches  to  the  0-switches  and  then  it  enters  the  motor,  which  indicates  what  it  does...and 
say  that  that’s  an  indication  [that]  mechanical  energy's  being  produced.  (Appendix  I, 
statement  46) 


Conclusions 

The  process  of  design  is  both  data-diiven  and  goal-driven,  and  both  utilizes  and  is  shaped 
by  several  sources  of  knowledge  and  information.  Our  discussion  of  instructional  problem  solving 
is  similar  to  other  information-processing  analyses  of  design  problem  solving,  involving 
differentiated  problem  subspaces  and  multiple  functional  roles. 

This  analysis  of  instructional  design  by  relative  novices  extends  the  information-processing 
literature.  Most  of  the  previous  analyses  of  problem  solving  in  tasks  of  arrangement  and  design 
either  used  highly  constrained  tasks,  such  as  cryptarithmetic  (Newell  &  Simon,  1972)  and 
planning  a  sequence  of  errands  (Hayes-Roth  &  Hayes-Roth,  1976),  or  focused  on  processes  that 
require  a  great  deal  of  domain-specific  expertise  (Jeffries  et  al.,  1981;  Pirolli  &  Berger,  1991: 
Stefik,  1981;  Ullman,  et  al.,  1988;  Voss  et  al.,  1983).  In  our  study,  the  relatively  open-ended  and 
large  task  of  instructional  design  was  perfonned  by  relative  novices  who  had  only  a  few  hours  of 
formal  training  in  instructional  design  (for  constructing  lesson  plans)  and  had  less  than  two  hours 
of  study  in  the  subject-matter  domain  of  the  VST20(X).  The  framework  that  we  used  successfully 
to  interpret  our  dau  is  generally  consistent  with  the  results  of  previous  research,  and  therefore  .idds 
a  segment  of  the  scientific  problem  space  to  which  the  information-processing  approach  has  been 
applied. 

The  quasi-social  nature  of  our  characterization  is  metaphorical,  of  course,  but  it  facilitated 
our  understanding  and  interpretation  of  our  data.  We  find  it  intriguing  to  speculate  that  the 
metaphor  of  social  interaction  may  have  a  substantive  basis  for  the  analysis  of  individual  cognitive 
activity.  Most  complex  cognition  occurs  in  settings  of  collaboration,  and  the  patterns  of  activity 
that  people  learn  in  those  settings  could  well  shape  the  processes  of  problem  solving  that  we 
engage  in  when  we  work  as  individuals. 
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Appendix  I 

Actors  Participating  in  First  Half  of  Sally's  Design 

The  table  below  shows  the  actors  who  participate  and  the  actions  each  performs.  Senior  managers 
are  indicated  in  bold-face,  middle  managers  in  plain  typeface,  and  builders  in  italics.  Action 
numbers  in  the  third  column  refer  to  the  numbaed  procedure  from  Table  1  in  the  main  body  of  the 
paper,  that  each  actor  seems  to  be  performing. 


# 

Statement 

Actor  (Action  #) 

1 

How  would  I  break  it  down  to  teach  something  like  that? 

Facilitator  (2) 

2 

...  are  they  going  to  use  the  computer  at  all,  or ...  are  they 
going  to  be  learning  to  operate  this?  The  same  thing  1  did.  It’s 
not  the  actual — 

Architect  (3) 

3 

Are  they  going  to  have  it .  .  so  you  could  have  them  actually 
doing  stuff? 

Architect  (3) 

4 

Then  I  just  orally  tell  you  how  I'd  set  this 

Architect  (1) 

5 

1  guess  it  doesn't  matter  what  I  tell  you,  but  it's  good  to — 

Architect  (1) 

6 

Yeah,  as  I  do  it.  not . . .  just  any  idea  that  I  come  up  with. 

Architect  (1) 

7 

I'm  used  to  doing  a  presentation  after  I've  thought  about  it 

Memory 

8 

Present  it  as  I  do  it.  Not  present  it,  but  as  I  think  about  it. 

Architect  (I) 

9 

I  think  what  I  would  do  first  is  give  them  a  general  overview  of 
what  it  is. 

Iniermediate  Builder 

10 

Basically  son  of  follow  how  I  learned  to  do  it,  I  guess. 

Abstract  Builder 

11 

Tell  them . . .  what  it  does,  the  general  purpose  of  it,  and  why 
it's  unique. 

Intermediate  Builder 

12 

...  I'm  contemplating  whether  it  would  be  worth  [it]  to 
demonstrate  it  to  begin  with  or  not 

Speculator 

13 

Just  to  get  them  to  see  what  it's  doing  before  I  stan  describing  it 

Advisor  (2) 

14 

And  that  I  haven't  decided  about. 

Speculator 

15 

But  I'll  continue  with  the  overview  idea. 

Facilitator  (1) 

16 

So,  I'd  give  them  the  general  introduction  about  what  it  docs  and 
why  it's  so  special. 

Intermediate  Builder 
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17  and  of  course  the  thing  might  be  there.  I'd  be  showing  them  the 
thing. 

1 8  I'm  thinking  now  about  whether  I  should  tell  them  about  the 
differences  between  the  wires. 

19  I  think  I  would  tell  them  the  difference  between  the  two  kinds  of 
wires  to  emphasize  the  fact  that  it  needs — it's  important  to  know 
from  the  beginning  that  it  has  to,  like  a. car,  it  ne^  a  source  of 
its  own  power  to  get  going  just  before  it  starts  energy. 

20  So  they  can  sort  of  differentiate  [the  types  of  ener^]  and  that 
will  help  them  later  on  in  understanding  scvoal  things  that  they 
need  to  do,  about  like  setting  up . . .  how  you  can  charge  one 
thing  while  you're  doing  the  other,  and  get  energy  for  the  overall 
machine  while  you're  doing  something  else. 

21  So,  differentiate,  describe  the  different  types  of  wires  and  how 
there's  electricity  in  it 

22  Going  back  to  the — 

23  Before  that,  I  would  de  cribc  the  three  kinds  of  energy  it  has  in 
each  of  the  uniL«  x  obtain  the  different  kinds  of  energy — the 
Tablograph,  the  Vegetor,  and  the  Impulse  Purifier. 

24  OK,  those  are  the  three  units ...  the  three  kinds  of  things  that 
they  would  produce  and  the  different  kinds  of  energy — just  give 
them  an  introduction  to  that. 

25  I  m  not  thinking  anything  now — it's  sub-brain  right  now. 

26  I  guess  I  would  provide  them  with  a  structure,  maybe  to — 

27  ...  all  this  information  I'm  giving  them, . . .  I've  learned  it  in 
all  my  classes  recently. 

28  ...  Give  them  a  table,  and  have  it  set  up  for  them . , .  one 
column  for  each  unit,  what  kind  of  energy  it  produces,  what  it 
needs — ^source  of  raw  energy.  Fill  it  in. 

29  Now  I  have  to  remember  some  of  the  stuff  that  I  learned. 

30  I  would . . .  well,  the  converters  are  pretty  much  tlic  same. 

31  Starting  with  this  unit,  [the]  source  of  raw  energy  (pause) 

32  I’m  trying  to  remember  what  these  things  are  specifically 
called — the  TaO. 

33  I  knew  I  had  to  use  them, 

34  but  maybe  it  might  just  be  a  source  of  confusion  of  I  tried  to — 


Intermediate  Builder 
Speculator 
Detail  Builder 

Advisor  (2) 

Intermediate  Builder 

Facilitator  (1) 
Intermediate  Builder 

Facilitator  (3) 

Monitor  (2) 
Abstract  Builder 

Memory 

Detail  Builder 

Monitor  (1) 
Memory 
Detail  Builder 
Architect  (3) 

Memory 

Advisor  (2) 
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35  Oh,  continuing  my  table: 

36  sources  of  raw  energy,  how  the  energy  is  extracted  for  each  one 
of  the  kinds  ...  the  needle,  scanner,  photoreceptor  . . . 

37  I  guess  I'm  not  going  to  worry  about  these. . . 

38  Okay,  that's  basically  their  table. 

39  Then,  I  think  this  helped  me  when  I  did  it: 

40  is  to  give  them  an  idea — and  this  is  when  I  would  demonstrate 
.  . .  one  of  them, 

41  probably  the  easiest  one,  the  most  complete  one. 

42  I  guess  this  one  skips  steps. 

43  but  just  fu-st  to  show  them  the  Tablograph  running.  Have  it  go 
.  .  .  [and]  you  get  a  "T"  in  here. 

44  Then  break  down  . . .  what  happened  and  just  let  them  see  that 
once. 

45  1  know.  I  don't  expect  iliem  to  understand  it 

46  But  I'm  going  to  give  them  this  flow  chart  I  did:  the  unit,  you  get 
the  raw  energy,  it  goes  to  the  convertor,  it's  impulse  energy,  and 
then  it  goes  to  the  I-switches,  it  goes  to  the  purifier — and  give 
them  this  little  flow  chart  to  follow.  And  then  how  it  goes  from 
the  1-switches  to  the  0-switches  and  then  it  enters  the  motor, 
which  indicates  what  it  does  . . .  and  say  that  that's  an  indication 
[that]  mechanical  energy's  being  produced. 

47  I  would  use  a  machine  and  show  them  once,  and  give  them  this 
more  streamlined  idea  of  what's  going  on.  Trace  as  I . . .  show 
where  each  [component]  is  on  here. 

48  I'm  contemplating  now  whether  I  should ....  in  my  general 
intro,  tell  them  atmut  the  parts  and  what  each  does,  like  what  the 
motor  indicates. 

49  I  don't  know  if  that  would  [make]  me  that  much  stronger  from 
the  beginning  or  not, 

50  but  so  they  get  the  idea  of  how  one  particular  one  flows. 

51  ...  actually,  at  this  point.  I've  gone  through  one. 

52  and  then  I  would  elaborate  on  each  of  these  components  . . . 
what  the  unit  is  and  what  it  does — converters, ...  the  I-switches 
. . .  orient  them  to  each  particular  component  on  the  flow. 


Facilitator  (1) 

Detail  Builder 

Detail  Builder 
Facilitator  (3) 
Memory 

Intermediate  Builder 

Advisor  (1) 
Memory 
Detail  Builder 

Intermediate  Builder 

Advisor  (1) 

Detail  Builder 


Intermediate  Builder 

Speculator 

Advisor  (2) 

Advisor  (2) 
Facilitator  (3) 
Intermediate  Builder 


Model  of  Design  Problem-Solving 
Korpi,  Greeno,  &  Jackson 


page  38 


53  So  that’s  when  I  would  do  it. 

54  I  think  that  it's  important  for  them  to  understand  how  it's 
flowing  first. . .  I  think  they'd  get  more  out  of . . .  each  one 
...  if  they  knew  what  it  was  doing  and  where  in  the  circuitry 
it's  doing  it 

55  So  orient  them  on  each  of  those  and  let  them . . . 

56  ...  I  don't  know  whether  I'd  have  30  computers . . . 

57  It'd  be  good  to  give  each  kid  a  feel,  to  try  to  move  the  switch 
...  as  I'm  orienting  each  one — to  play  with  each  one  as  they're 
going. 

58  Hopefully  that  would  be  possible.  I  guess  I'm  saying  what  I 
want  to  say,  so  I'll  say  it's  possible. 

59  and  they'll  get  to  play  with  them . . . 

60  Actually,  maybe  I'd  tell  them — 

61  I  guess  I  need  to  think  about ...  the  bottom  point  of  this. . . 

62  if  I'm  trying  to  (get  them  to  do]  critical  thinking  on  their  own  and 
(figure)  this  out  and  have  learning  be  a  process  for  them,  or  do  I 

. . .  just  want  (them)  to  be  able  to  (operate]  this? 

63  I  have  to  decide  what  my  focus  is . . . 

64  I'm  thinking,  I  could  be  totally  efficient  [and]  show  them  exactly 
how  to  do  one  of  them,  and  then  have  them  figure  out  the  other 
ones.  But  if  I  wanted  them  to  think — 

65  So  I  guess  it  would  depend  for  what  purposes  Fm  doing  it. 

66  I'm  not  sure  what  one  I'm  . . .  for. 

67  But,  if  I  had  to  have  them  well  trained  in  half  an  hour  because 
they  have  to  run  this  at  the  diesel  shop  that  afternoon.  I'd  just  not 
worry  about  the  learning  as  much. 

68  ...  that  would  be  something  to  think  about  before  I  did  it,  just 
why  or  how  I  want  them  to  learn  it. 

69  Will  I  want  them  to  learn  from  the  learning  or  [will]  I  just  want 
to  . . .  get  them  running  on  the  thing  . . .  have  them  play  with 
each  part?  ...  [is  it  a]  how-to  or  an  exercise  in  thinking?  Am  I 
doing  this  in  class  or  am  I  doing  this  because  I  have  to  do  it? 

You  know,  well  I  work  at  this  company  and  I  have  to  train  . . . 

70  I  guess  Fd  probably  want  them  to  think  about  it  anyway.  I'll 
assume  I  want  them  to  think  about  it 


Intermediate  Builder 
Advisor  (1) 

Intermediate  Builder 
Architect  (3) 
Advisor  (1) 

Architect  (2) 

Intermediate  Builder 
Intermediate  Builder 
Monitor  (1) 
Architect  (3) 

Monitor  (1) 
Speculator 

Advisor  (4) 
Architect  (3) 
Advisor  (1) 

Monitor  (1) 
Architect  (3) 


Architect  (2) 


Ontribution  Ust 


Or.  Pbillip  L  Acfccrmia 
UoivcnMy  ol  Mioacaou 
Depanment  ot  Piycbology 
7S  Bmt  Riwcr  Rmd 
N216  Ettou  Han 
Miooeapote.  MN  S54S3 

Dr.  Bccb  AddKO 
Dapartacoi  at  Paycbotogy 
Ruifm  Uotwiiiy 

mm 

Dr.  David  L  Altelon,  Code  131 
Navy  Pmoooii  RAD  Ccoier 
Sao  Diatom  CA  f213240OO 

Or.  Jotei  R.  AiMlarwa 
DepartBMDt  of  Pryefaotofy 
CanMgio>MattoQ  Uaiwanay 
Sefanlay  Pwt 
fasMbu^  PA  15213 

Dr.  Naaqr  &  Aadmoo 
Departaaol  of  Payctoiofy 
Uofvcntty  of  Maryland 
Colley  Part.  MD  VfH 

Dr.  Tbooiaa  H.  Aodaraoe 
Camcr  for  tbe  Study  of  Raadini 
174  Cbildm‘i  RoMarafa  Caotar 
51  Carty  Oriva 
Cbaapaipv  IL  41C0 

Dr.  Supben  J.  Andriole,  Chairman 
Cottcfe  of  InfomatioA  Suidiaa 
Drcnd  Unwaraity 
Phiiadeiphii.  PA  191<M 

Dr.  Grefory  Anrig 
Edueauonal  TaatMg  Samoc 
Prmoatoo.  NJ  0854] 

Tacbmeal  Oiractor.  ARI 
5001  EiacTiboaiar  Avanua 
Amdria.  VA  2Z333 

Or  Miebaal  E.  Alipood 

NYNEX 

AI  Laboratoiy 

Wlala  Ptain.  NY  10RM 

Dr.  Mciyt  S  Batar 

Navy  Panoand  RAO  Cemcr 

Sao  Dwyo,  CA  t215240OO 

Dr.  Hcinricb  BauerdaU  . 

EDM  der  Uawanitact 
Poatfad)  M40 

D-4800  Bidefald  1.  WEST  GERMANY 
FEDERAL  REPUBLIC  OF  GERMANY 

Dr.  haae  L  Bayar 
LmrScbooiliaiiiinni 

Smvkai 
Bob  40 

NcMooii.  PA  tOMM040 

Dr.  Mfoiifba  Dirihaua 
Bdurational  Tadiag 
Sarviea 

Piioeaioii.  NJ  tt541 

Dr  GauUMB  Bmm 
Dapanmem  of  Cooipiaar  Sbaocr 
Boa  IdiR  ScatiOR  B 
Vmdartdl  Univanity 
Nnb«0te.TN  37235 

Dr  Jote  Btect 
Taacban  Collage.  Bos  8 
CobHBtia  Uoivanicy 
525  Waal  I20ib  Siraat 
N«»  Yort.  NY  10027 


Dr.  fCaniMtb  R.  Boff 
AL/CFH 

Wn^t'Ptuanoo  AFB 
OH  45433^73 

Dr.  Jeff  Bonar 
Gtadaooa  Tacbnology.  loc. 

•00  Vsaal  Soaci 
Psubitfib.  PA  152U 

Dr.  Robert  Brama 
Coda  252 

Naval  Tuning  SyHaoa  Canur 
Orteodo,  PL  32824-3224 

Dr.  Aon  Btotm 
Graduate  Sdtool  of  PAiraboc 
Uoivaiity  of  Califoniia 
EMSr-4533  Tolmao  Hal 
Bertaky.  CA  bOJO 

Dr.  Jote  T.  Brutf 

Jaaea  &  McDonnea  Poundaboo 

Suite  HIO 

1034  South  Dtaoeaood  BM. 

Sl  Loim.  mo  43117 

I>.  Joanna  Capper 
3545  Atbemarte  Street.  NW 
Wadiio|»oo.  DC  »006 

Dr.  Rkbard  Catnaboee 
School  of  Pqrcbology 
GeorgM  ioaiiuse  of  Technology 
Atlanta,  GA  30332^170 

Dr.  Ruth  W.  Chafaay 
CDEC  Hamburg  Hall 
Carnegie  MeBon  Unnanky 
Pmabi^  PA  152U 

Dr.  Pad  R  Chatchar 

Parocptronici 

191 1  North  Pl  Myer  Or. 

Suiu  800 

ArteyoaVA  22209 

Dr.  Mkbalenc  Gbi 
Lrameig  RAD  Center 
UnbaniCy  of  Pitiabur|h 
3939  OTiwa  Straat 
PHttburgb.  PA  15210 

Dr.  Raymond  E  Chhdal 
UES  LAMP  Saaoee  Adviaor 
AUHRMIL 
Brooka  AFB.  IX  7S235 

Dr.  David  E  Oamaot 
Dapartmaoc  of  Piyebolofy 
Uoberaicy  of  South  Carabna 
Cehnbia.  SC  29208 

CM  of  Navd  Edueaboa  mid 
IViinBg  (N-5) 

NAS  Paamcoto.  PL  32508 

Dr.  Pad  Cobb 
Purdue  Uabmuty 
Buddmg 

W.  Ldayaoa,  IN  47907 

Or.  Rodney  Coating 
NIMK  Ba^  Brhavinr  and 
Co^acM  ScMBoa  fUaaanh 
5400  Fahan  Lane.  Rm  llC-10 
RoctvOa.  MD  20857 

rnmmanding  ORiem 
Naval  Bmearrti  Laboratory 
Coda  4827 

Wmtaigfecv  DC  20975-5000 


Or.  John  M.  Cornwell 
DepantDcnt  of  Piychology 
I/O  Prycbolo|y  Program 
Tuiane  Unbenity 
New  Orleana.  LA  70118 

Dr.  Mkhad  Coweo 
Code  142 

Navy  Persoond  RAO  Center 
Sm  Die^  CA  92152-4800 

Dr.  WaiiMi  Craoo 
Deparuaeot  of  PiycboloBr 
Toe  AAM  Unbenity 
Cnflrgp  Stabog  TX  77S43 

Dr.  Reiwwrti  E  Onw 

Atwpa  |f>g. 

P.O.  Bos  519 

SMia  Bartaca,  CA  93102 

CTBi/MoGraw-HiB  Libraiy 
2500  Garden  Road 
Mootcray.  CA  93940-5380 

Dr.  TVniaf 

Payefado^  Depanmeni 
Univanity  of  Aroona 
IW00Q.AZ  85721 

Dr.  Charles  E  Davis 
Educational  Tatting  Service 
Mail  Stop  2^T 
Prinectoo.  NJ  08541 

Marpret  Day.  LAnrian 
Appbed  S6e^  Asaodales 
P.O.  Bos  1072 
BuUm.  PA  140)9 

Dr.  Sharon  Derry 
Plorda  State  Unbenity 
Dcparuncni  of  Psychotop 
TaBahasaee,  FL  32304 

Dr.  Stephanie  Doane 
Unbenity  of  Illinois 
Oapartmant  of  Psycbolc^ 

403  Emt  Dadd  Street 
Cbampaiga.  IL  41820 

Dr.  J.  Stuart  Donn 
FaoAy  of  Ed*KaliOD 
Unbenity  of  Briliab  Coiuabia 
2125  Main  MaO 

Vancouver.  BC  CANADA  1Z4 

Dr.  Michad  DriOingi 
Bade  Reaeareh  OfDoe 
Army  Rmearcb  Inetiuite 
5001  Edanboaar  Avenue 
AlcMidha.  VA  ZZ333 

Dr.  PierTa  Dipid 
Orgaadabon  for  Eoooooac 
Coopencioe  and  Dmiopmcni 
Z  rue  Andra-Paecd 
75014  PARJS 
FRANCE 

Dr.  Richard  Duran 
Oraduatf  School  of  EAwaticio 
Unbenity  of  Caiifonita 
Sanu  Barbmi.  CA  93104 

Dr.  Nancy  EMredge 
CoBege  of  Educauoo 
Dbidon  of  Spedd  Educauoo 
Tbe  Unbenity  of  Ahstona 
'nieaoci.AZ  85721 

Dr.  John  Ellis 

Panonnd  RAD  Center 
Code  15 

Sen  Diego.  CA  92152-4800 


Dr.  Sumo  Eabrottoo 
UoMn^  of  Kaons 
Pi)fcfaolo|y  DapirtMoi 

Uomot.  KS  <604S 

Dr.  Suno  Efiain 
144  S  Mfinnin  Awnue 
MMte.  NJ  ma 

ERIC  Puttlji  Armioilinni 
2440  RMMfa  BM.  Suiio  400 
RocfcvOc.  MD  »8SO>3238 

Dr.  K.  Aodm  Eriatoa 
UiwMury  of  Cotorado 
DcpwtdMoc  of  Pij^ciioto^ 
rimpiM  001345 
BouUff.  OO  nSOMMS 

Dr.  Mactho  Ends 
Dept  of  Coaptwy  ScMwe 
Oiooii  loMiiule  of  ToGboolo0 
Owopp.  IL  40014 

Dr.  LomiM  D.  Eyde 
US  OfPee  of  Pawonod  Minaymmi 
Offiee  of  Peraaood  Rjacawfa  aod 
DeadopaMot  Copawnt 
llOOESu  NW 
WMbnim  DC  30415 

Dr.  Beotrict  X  Parr 
Atmf  Rmatf±  loMibilo 
PERl-IC 

5001  Rimhna»ar  A%ooiia 
Akmdria,  VA  22333 

Dr.  Manbal  J.  Parr 
ParT'Sigbt  Ga 
2520  North  Vorooo  Straec 
ArlM^un.  VA  2Z207 

Dr.  P-A.  Faderioo 
Code  51 
NPRDC 

Saa  Diat^  CA  tZUlntfOO 

Dr.  Richard  L  Parpm 
Aaariran  CoOe|a  TaMiof 
P.O.  Boa  148 
loua  Qty.  lA  52243 

Mr.  Walaoa  Poiineig 
Editcaiinnd  Tachoolop 
Boh  Bcraock  A  Nw>wao. 

10  Moukoo  St 
Caobridfe.  MA  02238 

Dr.  Vkiot  Fiddi 
Paparf  ml  of  Piychotop 
MooiirwBaiy  Coltp 
RocMe,  MD  30850 

Dr.  Mkhad  Pteaigao 

Code  14 

NPRDC 

SnDiapxCA  021524800 

Dr.  1  D.  Platehar 
Imtibka  for  Drfne  Anaiyafg 
mK  niiiaiprd  St 
AJoMdr^VA  22311 

Dr.  Lioda  Plover 
Canwpt  MdlfW  Uaiwaiiky 
Daparweni  of  Pni^kh 
Pinaburih.  PA  152U 

Dr.  Cart  a  Pradcrikaen 
DapL  ofSducaciOQd  Pipcfaolopp 
M^d  Univaniiy 
3700  McTavhb  Straac 
Momrad,  Ouaboc 
CANADA  H3A  m 


Dr.  Nonoao  Praderikaco 
Educational  TaMiOf  Sccvwe 
(05.R) 

Prickoaioo,  NJ  08541 

Dr.  Alfred  R.  PregK 
APOSR/NU  BUg.  410 
BoOog  APR  DC  303324448 

Dr.  Marfl  P.  Garmt 
Diraccor  of  Cogpitac  Sdcaoe 
Dapartaicot  of  Piychoiogy.  Rooa  312 
UowaidQf  of  Aracxia 
'nicaoo,  AZ  85721 

Dr.  Dadra  Gaatocr 
Nortbacuere  UoMciay 
Departfneot  of  Pqichol^ 

3029  Sberidao  Rood 
Sadi  HaO,  Rm  102 
Bmtoo.  0.  40208-2710 

Chair.  OapactoMot  of 
Cootputer  Sekoee 
Oaorye  Maaoo  Uowonity 
PairCai,  VA  22030 

Dr.  Hden  Gi^cy 
Natfonal  Sekooe  Pouadaiioo 
1800  G  Sc..  N.W..  Rffl.  310 
WMhiopoo.  DC  30550 

Dr.  PhihpCiflk 
ARJ'Port  Gordoo 
ATTN:  PERI-ICD 
Port  Gordoo.  GA  30905 

Dr.  Hachert  Gkahurf 
Bos«4 

Taachera  CaOege 
Cotunbea  Uokmicy 
525  Wcd  Ulal  Send 
Nev  Yort  KY  10027 

Or.  Draa'  Gkoocr 
Educatiorial  Tcdiog  Service 
Prinoeum,  NJ  08541 

Mr.  UeGtadwia 
305  Davia  Avenue 
Ladhurp  VA  22075 

Dr.  Robert  Okaer 
l  ifTiint  Reaeattfa 
A  Devaiopaent  Ccoter 
Unkuniiy  of  Pictebuegb 
3939  O’Hara  Soacc 
Piuaburgh.  PA  15360 

iir.  Maevk  D.  Glocfc 
101  HoMdoad  Tarraee 
hteca.  NY  14854 

Dr.  Sob  Okutabwi 
DapaetBaol  of  ftpchologp 
Priooeloa  Uokac^ 

Prkeeioo.  NJ  08544-1810 

Dr.  Suaao  R.  OoUbmo 
Paabody  Colege,  Bok  45 
▼aDovaat  vwrwy 
Naabidk,lN  37303 

Mr.  HaroU  OoldMck 
UoMniiy  of  DC 
DipactBBK  Qv8  Pnginaaring 
Bldg.  42.  Roob  12 
4200  Caaoadkui  Avcouc;  N.W. 
WtfMi^oa.  DC  3B008 

Dr.  T.  Gododarpi 
Gaorgk  lodicuia  of 

School  of  Induatrid 

and  SydaaM  Enginanmg 
AUanu.  CA  303324205 


Dr.  Mardyo  R  Goakg 
Ofltoe  of  Penonnd  RAD 
1900  E  Sl,  NW,  Rooa  4462 
Office  of  Penoond  Maoageacot 
Weebiogtoo.  DC  30415 

Dr.  Arthur  C  Graeeaer 
Department  of  Piychdocr 
Mmftiii  State  Uukmi^ 

Meaybii.  IN  36152 

Jordui  GrafBao,  PIlD. 

CogpiowB  Ncuroaskoce 
Seeboo.  Medkd  Ncurokv 
Braoch-NINDS.  Bkg  \<K  Ra.  5C422 
Bdhaik.  MD  30892 

Dr.  Wayne  Gray 
Oraduaic  School  of  Educaboo 
Pordbaa  Uokcniiy 
113  Wad  40th  Street 
New  York.  NY  10023 

Dr.  Bert  Giacs 
Jobm  Hopkioa  Uananily 
Depattment  of  Piychology 
Chartee  A  34tb  Street 
Baiima.  MD  21218 

Dr.  Jaaea  G.  Oracoo 
School  of  Educatioo 
Stanford  UokcciiQr 
Rooa  31) 

Stdrford.  CA  44305 

Dr.  Stephen  Graaabcrg 
Center  Cor  Adaptive  Syiteae 
Rooa  244 

111  Cummington  Sttact 
Baetoo  Uoiveni^ 

BoatoaMA  QQ215 

Naval  Ocean  Syatena  Cmier 
rnaaand  Sup^  Technology  Dividoo 
Ann:  Mr.  J.  Groaaaaoi,  Code  44 
Bldg.  334 

SaeE>kgo.CA  9215^5000 

Dr.  Mkhad  Habco 
DORNIER  GMBH 
P.O.  Boa  1420 
D-TPOO  Prkdrkhabafco  1 
WEST  GERMANY 

Dr.  Henry  M.  Hdff 
Halff  Raaoureaa.  loc. 

4918  33rd  Road.  North 
Arksguai.  VA  22207 

Marilyn  Halpam,  Ubneian 
Brighaa  Library 

Teding  Service 
Caar  aod  Roaadak  Roadi 
PrBeeloii,NJ  08541 

Dr.  Ivar  Haoico 
Putaarata  Piykologi^eoede 
AfcantaudOilo  ad 
•0l5Oalo  1 
NORWAY 

Dr.  Stephen  J.  Hanaon 
Laming  A  Knoa4adge 
Aupadrinn  Raaoarrh 
Skaena  Raaeaceb  Canter 
755  Colkfe  Road  Bad 
Pnwdooi.  NJ  08540 

Suvan  Hamad 
Edhor,  The  Debaviord  and 
Brain  Seknea 
30  Nmu  Stract  Suite  240 
PrvKCtoa  NJ  08542 


0903/91 


Dr.  Drli^yo  HaruMcfa 
Unverti^  of  Ittnoa 
SI  Otrty  Drive 
ChMBpiiin.  IL  01S20 

Janee  Hart 

Drpanaum  of  (be  Nai^ 

Joist  CALS  Manapnwnt  Office 
SIW  Lwburt  P>c 
SlyiiDC  4  Rora  701 
P*aHRb.VA  22M1 

Dr.  RaidHaMie 
UsiveiaiQf  of  Coteado 
Daparweni  of  Paycfaotop 
BoMdar.  OO  «D0MH4 

Dr.  Bactan  Hayaa  Roth 
KsoaMfe  Syatn  Labonloiy 
Scaoford  Usiveniiy 
701  Wdcb  Rood.  BM^  C 
faloAluxCA  KMM 

Dr.  Par  HatOMncs 
UoMoniiy  of  Oak> 

usrr 

Ba  10S9 

03M  Olio.  NORWAY 

Dr.  Sanaa  Hkbcrt 
Dapartnast  of  PAirarinnai 
OmolopnaM 
UoMoreity  of  Dalaotra 
Naoact,  DE  197U 

Dr.  KcfCb  Ho^oak 
DapartaMOt  of  Piycbolofjf 
UoMoraity  of  Califamia 
LaaAntBte.GA  00024 

Prot  Lua  P.  Hoatikc 
Isatiust  fur  Paydbotogja 
RWTH  Aacben 
Jatgaracraaoa  17/10 
D-5100  Aacbco 
WEST  GERMANY 

Ma.  Jtdia  S.  Hot^ 
ranhridfr  Ussanity  Praaa 
40  Waat  aUi  Siraac 
Nav  Yoft.  NY  lOOll 

Dr.  wate  HomI 
Chief  Sdcoiiat 
AFHRLCA 

Brooka  APB.  IX  78Z3S.S401 

Dr.  Stavas  Hunka 
3-104  Educ.  N. 

U  Bivanity  of  AJbarta 
Ednomoo,  Alberta 
CANAI3A  TiCXiS 

Dr.  E«4  Huh 
Dapc  of  PiychoioQf,  N1-2S 
Uaivaraity  of  Waahis^oa 
Saouia.  WA  0010$ 

Dr.  Oioffio  Ispripala 
Conpuiar  Sfianot  DopartnaH 
Tanpla  Unwniiy 
PMidilpOh.  PA  MZ2 

Dr.  Mntn  J.  Ippal 
Ceniar  for  the  Study  of 
»««*  mtA  isainicOoB 
Laidan  UaNanicy 
P.  a  Bos  9S5S 
OOORBLaidoB 
THE  NETHERIAKDS 

Dr.  Robert  Jassarooe 
Elac  asd  Conptsar  Es^  Dapc 
UoNsnity  of  South  Carabsa 
Cobinbia.  SC  2WD8 


Dr.  Claude  Janvier 
Uoiveraite'  du  Ouebac  a 
Mootraal,  QRADE* 

P.  O.  Bos  M84  auoe  A* 

Montreal,  Ouebac  H3C  3Pft 
CANADA 

Dr.  Robio  Jaffiiaa 
Hesteu-Paekard 
Laboniorka.  1U*17 
P.a  Boi  1M90 
PaloAkn.  CA  00034060 

Dr.  Bosne  E  Jobs 
DcpHimcH  of  Conpuiar  Sciaoee 
Camgie  MaSos  UuiiaraiQf 
SOOO  Poebaa  Avesue 
Pkttbuc#i.PA  152U 

Dr.  Daoid  E  Jooea 
US  Nuekar  Rrjdatory 
ConniaatoQ 
NRR/12  B4 

WnfanmxuDC  20S55 

Dr.  John  Jooidea 
Departnent  of  Pqrchoi^ 

UoMenity  of  Mkbigui 
An  Arbor.  MI  48104 

Dr.  Marcel  Juat 
Carae^McOon  Ualvmity 
DepartsMot  of  Payebotofy 
Sebeniey  Pait 
Pioaburip.  PA  1S2U 

Dr.  Ruth  Kanfrr 
UfMvaniQf  of  Mionaaou 
Depattnem  of  Payefaoiofy 
Elboa  HaB 
TSERNacRnd 
Mima^ba,  MN  554S5 

Dr.  A  Ksreakiff-Saaih 

MRCCDU 

17  Gordon  Stract 

ENGLAND  WCIH  OAH 

Dr.  MiumSKao 
PSC  ffiZ  Bos  D 
PPOAE  00490.1300 

Dr  .  Wendy  Uoa 

DM  T.  J.  Wataon 
Raaaartb  Or. 

P.a  Ejstm 

Yorttoan  Haiphia,  NY  10300 

Dr.  Jaffery  L  Ksonin^on 
Sdwol  of  Engr.  A  Appbad  Sorocaa 
Souibare  Mathodut  Uisvataiiy 
DaMa^lX  73273 

Dr.  Jarsary  Ppatrick. 

Dapartnani  of 
MaibaMMa 
NS  AdarboM  Hal 
UoMnity  of  Gaorgio 
Attann,  GA  30f0Z 

Dr  Wakar  Kintach 
OapannaH  of  Piychofosi 
UHwarasy  of  Colorado 
BouMar,  00  I03004M5 

Mnri  Usdlraw  ~ 

Syatani  Caniar,  Code  2212 
Budding  lITl/i 
Nanpoit,  R]  02S41 

Dr.  Janat  L  KHoAior 
Osorgia  Isatsutc  of  Tacfarwlop 
Col^  of  Conputing 
Aitanta,  GA  303324200 


Dr.  Richard  J.  Koubek 
School  of  Induatrial 
Engineering 
Griaaon  HaB 
Purdue  Univaraity 
Waac  L^ayatte.  IN  47907 

Dr.  Patrick  Kylooan 
AFHRUMOEL 
Brooka  APR  7X  78235 

Richard  iHScman 
Cfaisnanilinf  (G-PWP) 

US  CoHt  Guard 
2100  Second  St,  SW 
W^^oo,  DC  205934001 

Dr.  niMkai 
Cmegk'MaOon  UaNanity 
D^HtneH  of  Paydtolofir 
Phttburgh,  PA  132D 

COMMANDER  USAR] 

PERJ-BR  (ATIN:  LaMoo) 

3001  namhcraTT  Avenue 
AkBndin.VA  22333-3400 

Dr.  Yub%}ang  Lac 

Depertment  of  ConptSff  Science 

CodeCSAE 

Naval  Postpaduaic  School 
Monterey.  CA  03043 

Dr.  Paul  E  Lchoer 

Sye.  Engr. 

Gaorge  Maaoo  UnivaniQr 
4400  UnivaiBiQi  Drb« 

PwUkVA  22030 

Dr.  Riebazd  Lab 
Educational  Tatiog  Service 
PriocstMt  HI  08341 

Dr.  Marcia  C  Laon 
Otaduaie  School 
of  Educatinit  EMST 
Tolnan  HaB 
Uaivaewy  of  Cohforaia 
Bartalcy.  CA  04720 

Logieoo  Inc  (Attn:  Ubrvy) 
Tacxieai  and  TMning  Synen 
DMaion 
P.a  Bos  85138 
Sm  Diepo^  CA  92138-3158 

MinR  Dr.  CfariaiiaD  Lobauaer 

Maria  Thrrairn  raarm^ 

1130  Wien 
AUSnUA 

LT(N)  CDF.  Ipoo 

Cenannd  Pvaoonal  Applied 
Raaarth  Coordioaie 
Markine  rrtnnand  Haadqunrtsn 
PMO  Hslifa^  RSh  B3K  2X0 
CANADA 

Van  M.  Malac 
NPRDC  Code  142 
SnDia^CA  921524800 

Dr.  Jane  MNo 
idaaCodcER22 
NASA  Johnaon  Space  Canter 
HeiMioo.TX  77038 

Dr.  WHmd  L  Maloy 

Codebt 

NETPMSA 

Pnaemh.  PL  323093000 


Dr.  Mary  Maiteo 
Dirvaonu 

laatniGMioal  Tadmotogy 
HQUSAFA/DFET 
USAF  Acadcny.  00  «)e4a-5017 

Dr.  SMidn  P.  MkiImM 
Dept  of  Pyfcology 
Seo  DiegD  Suu  Uoweniiy 
SmDmtfKCA  92m 

Dr.  OancD  X  Umtm 
Head,  DoD  Coordiaator. 
Ilacnnint  Plaoi  aod  Pwyi— 
BcwMfaCode  PERS2FP/Z34 
Nwy  Aodo.  Rood  202 
WaetetfflitDC  30030 

Dr.  PHtaherfa  Urntm 
AL/HRA,Slop44 
Wittaae  AFB 
AZ  0240 

Or.  Jota 

Tacfaoksal  CoBBUDieatioQe 
ProfTiB 

ZMOBoMUd  BML 
UoMrMQr  of  Mkfaipo 
Am  After.  M!  48109 

Dr.  RkM  e.  Miyv 
DepadBOM  of  P^FCfaolop 
UoMraiQr  of  CelifonMa 
Santa  Bartwa.  CA  01(» 

Or.  Oted  J.  MoOuiofMM 
Oaflaudet  VtmmnUjf 
800  Ploridi  Aranye,  N.E 
Waateiffoo.  DC  2D0Q2 

Dr.  Kathlean  MdCooMi 
CoteBtei  Unwtreiry 
DepansKiM  of  CoeepMtar  SoiencK 
450  Coapuiar  Seicnee  BMdrting 
Hm  York.  KY  10027 

Dr.  Mietetf  McNom 
DETa  ALVCFHI 
BUX;  248 

Wr^.PaOMa  APB.  OH  43432 

Dr.  Si%  Miincke.  PhJX 
PofBarau  Caoiar  for  Ladankab 
ChneciaoaliBae  Vcidgade  8 
1424  loteteavD  K 
DENMARK 

Dr.  Alan  Meytoadi 
NmI  riMfch  LatanNory 
Code  5510 

4SS5  OMriook  Ava..  SW 
Wate^Mso.  DC  30973-5000 

Dr.  Joal  A  ftficted 
Oepadacoi  of  Ptayiiolo|Qr 
Riiib  PrafeyuriofrSl  Lidtae 
Madied 

Madkal  CoB^ 

Clacapx  0.  «06U 

Dr.  CMiintM.  Mdctel 
Scteel  of  Mut  andSyt  Eog. 
Caoior  for  Mm-MacMne 
SftumB  RanarB 
CaoriM  Intecuto  of  Tactanoloer 
Ailaaia.GA  385324205 

Dr.  Ben  B.  Morgen,  Ir. 
DepaRaaM  of  ftycBolo^ 
Uateairy  of  Cawtrai  Florida 
Ortedo.  PL  328104390 


Dr.  Randy  Muoiaa 
Human  Scicnoea 
Waatiogteuac  Scicooe 
A  Tactoolo0  Or. 

UlO  Beulah  Road 
PkuburiB.  PA  D235 

Dr.  Alkn  Muoro 
Behavioral  Tachoolofif 
Lateratorioi  •  use 
250  H.  Hater  Dr.,  Sihia  309 
Radoodo  Baach,  CA  90277 

Chair.  DopartmaDt  of  Waapooa  aod 
Syabema  Pn^aaring 
Naval  Aeadc^ 

Aaoapote  MD  21402 

Diatributaon  Support  Damioe 
Bureau  of  Naval  Paraoooci 
PERS471D 
WmtiioimQ.  DC  20370 

Aradmiir  Pregt  A  Raaaarch  Branch 
Naval  Tacteieal  Teaming  Command 
Code  N-42 
NAS  Mcopba  (75) 

Mining  TN  30834 

E>epuQr  Diraeior  Manpoaar, 

Peraoond  aod  Tnioiog  Div. 

Naval  Sea  Syetema  Command 
ATTN:  Code  04MP  311 
Waabingum.  DC  20342 

Nmy  Supply  Syetema  Command 
NAVSUP  35U 
ATTN:  Sandra  Bordeo 
Wmbmi^DC  3074-5000 

Mr.  ].  Neteaen 
TWnte  Uofvenity 

Fac  BtbL  Toegepaate  Onderwyakuadc 
P.  a  Boa  217 
7500AEEmctede 
The  NETHERLANDS 

Dr.  Raymond  &  Nkkcraoo 
5  Clem  on  Rood 
BadTord,  MA  01730 

Dr.  Ndi  J.  NiaMo 
PfpfftiMwt  of  Computer  Soenem 
Sia^ocd  Unamity 
SCHfoed,  CA  90034060 

DiPBOor 

Training  Taefanoloy  Dapartment 
NPRDC  (Code  D) 

Sm  Oia»a.  CA  921524800 

Ubracy.  NPRDC 
Code  041 

Sm  Diegos  CA  92t524«)0 
Ubrariao 

Navy  Caour  for  Appiod  Ramarch 
to  Artifkial  hWfligmnr 
Naval  Rmeatiii  Latentoty 
Code  3510 

INaate^foo,  DC  20973-5000 

Special  Aaojeraw,  for  Rmiiarrti 
Management 

CM  of  Navy  Paraoony  (PERSOITT) 
Deportment  of  the  Nmy 
WmbtogtOQ.  DC  3a35A20OO 

Dr.  iamc  Oriuaky 
laociuiu  for  Defcnae  Analywa 
1»1  N.  Boauragud  Sc 
Akimndrto.  VA  22311 

Dr.  Olann  Oep 

NOSC  Code  441 

San  Diegos  CA  921524800 


Dr.  Otcfaooo  Park 
Army  ReacaiUi  Inatkute 
PERl'2 

3001  Eiambonrr  Avenue 
Almdria.  VA  22333 

Dr.  Rjy  &  Pers 
ARI  (PERl-D) 

3001  Eiamhcmcr  Avenue 
AlMdte.  VA  22333 

CV.  (MD)  Dr.  Antonio  Peri 
Cbptaio  riNMC 
MafM«  VJ>.C.  y  So 
MINtSTERO  DIFESA  -  MARINA 
80100  ROMA  -  FTALY 

Dr.  Naoqr  N.  Perry 
Navy  Eduraiioo  aid  Tnioiog 
Profram  Support  Aaivify 
Code-047 

2435 

PoBoaeato,PL  32509-5000 

CDR  Fnnk  C  Petho 
Navy  Poatgraduaie 

aCoOOi 

Code  OR/PE 
Mooicray.  CA  93943 

Dr.  Pour  Pirott 
School  of  Bducaiioc 
Univiniiy  of  Califonia 
Boketey,  CA  94720 

Dr.  Manba  Pohoo 
Department  of  Piycbology 
UnhaniQf  of  Colorado 
Boulder.  (X>  809090944 

Dr.  Peter  Polaoo 
Uoivaraiiy  of  Colorado 
Department  of  Pyefaotogy 
Boulder.  00  n9094D44 

Ihqrc  Info  -  CD  aod  M 
Amerkao  ftydwiogicy  Aaaoc 
1200  Ubie  Straet 
Aite|h».VA  22201 

Mr.  Pater  Purdue  (55Pd) 
Opcrateo*  Raminii 
Navy  Poatgradualc  School 
Montcray.  CA  9390 

Dr.  Stophao  Radar 
NWREL 

101  SW  Man.  Suite  300 
PortMOR  97304 

Dr.  I  Waalcy  Ragan 
APHRUIDl 

BroQkaAPB.TX  7805 

Dr.  Daniy  Reahrrg 
Read  Co8egt 
Dapartment  of  Paythoiogy 
PenhBd.OR  97382 

Dr.  Brian  Rater 
Dapartment  of  Piychoto0 
Giam  Han 
Prinaaton  UnteraiQf 
Prtoeaion.  NJ  88340 

Dr.  Lauren  Reanick 
Imnmg  R  4  DCantm 
Untetii^  of  Ptoaburg^ 

3839  OHara  Straa 
Pitabitfib.  PA  D2D 

[>.  (ritoert  Rican) 

Matt  Stop  K01.14 
Grumman  Araaft  Syatema 
Bahpar.  NY  11714 


09^1 


Dr.  EdwiM  L  Riniaod 
Dcpc  of  Oagyutw  Mid 
lafotnMioo  ScMPot 
UoMniqr  of  MaHMbuMCu 
AiBteM.  MA  91003 

Mr.  W.  A.  Rn> 

HmM.  Hummi  Pmioci  DMiioo 
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